Summary: Bigger isn’t automatically better: there’s no bad model, only a model that hasn’t been taught the task. Before upgrading, narrow the task, reframe the question, and manage context, which often lets a small, cheap, fast model do the job. Choose on your evals, not public benchmarks: start with the biggest model, then find the smallest that meets the KPI. Plan with a strong model and execute with a cheaper one, and keep prompts tuned per model version. Whether to build model-agnostic (swap per component, keep a tested fallback) or commit to one lab’s models is contested.
Key ideas
- A ~35B Qwen model insisted a car in a video wasn’t changing lanes; a big model got it right but at negative ROI. Asking instead “does the line move from the left of the frame to the right?” made the small model correct (ai-engineering-podcast—why-your-llm-costs-so-much, sharon-dahan).
- Scheduling PDFs failed even on Opus; a small model found cut points, the PDF was sliced, and the slices worked “phenomenally” (ai-engineering-podcast—why-your-llm-costs-so-much).
- For ~30B models, target an “attention-deficit” level of focus: no more than 2–3 logic steps per call (ai-engineering-podcast—why-your-llm-costs-so-much).
- Newer models need less role-setting preamble, and preamble eats context in small models (ai-engineering-podcast—why-your-llm-costs-so-much).
- Reasoning budgets: many inference stacks let you cap thinking tokens; budget them so reasoning doesn’t consume the whole context (ai-engineering-podcast—why-your-llm-costs-so-much).
- Batch (“take one”) jobs of ~1.8M prompts must work the first time, so prompt engineering matters even more (ai-engineering-podcast—why-your-llm-costs-so-much).
- B2C: one conversation can’t cost the user’s lifetime value, so stay on small models (ai-engineering-podcast—why-your-llm-costs-so-much).
- Consumer apps still make users choose models: partly marketing and psychology (launches like “Fable 5”); automatic routing via classifiers is coming (ai-engineering-podcast—why-your-llm-costs-so-much).
- His coding defaults: Sonnet/Opus; Fable only for specific, patience-worthy cases; reasoning-heavy expensive models can over-research simple asks (ai-engineering-podcast—why-your-llm-costs-so-much).
- Every product should have a primary and a pre-tested fallback model (downgrade a version, or switch to another vendor’s family) (ai-engineering-podcast—ai-infra-at-scale, dor-cohen).
- Developers must know their models well enough to tune prompts per model version (ai-engineering-podcast—ai-infra-at-scale).
- Public benchmarks often don’t match your environment; test on your own evals (ai-engineering-podcast—ai-infra-at-scale).
- Gavriel Cohen follows Anthropic’s view that agents aren’t portable across providers: each lab’s models behave differently, so he didn’t build model-switching (langtalks—71-claw-architectures).
- Start with the biggest model, then find the smallest that meets the KPI: NVIDIA’s open Nemotron (~33B active) matched Claude Sonnet 4.5 on CVE enrichment at a fraction of the cost; ~70B models suffice to flag crypto miners (langtalks—66-scaling-llmops, avi-lumelsky).
- Small models sometimes grasp a narrow task better (e.g. spotting repeated base64 encoding as suspicious) (langtalks—66-scaling-llmops).
- Beyond picking among API models: training a small task-specific model can beat a general LLM on accuracy and speed; see small-language-models (langtalks—59-slms).
- Niche models: some aim to be best under real-time latency, others at long context or specific tasks. Per-intent compute (as GPT-5 varies thinking) may come to inference stacks. Smaller models gain ground as ROI starts to matter (langtalks—58-reinvent-predictions).
- For copywriting (incl. Hebrew), the host finds GPT 5.6 the best; Fable is “genius but hard to talk to” (ignore-instructions—28-ai-marketing-enso).
- Model-agnostic by design: every component (builders, runners) can use Gemini, OpenAI, Anthropic, or xAI behind a generic interface; an ML team keeps benchmarking and swapping per component. A better open-source voice-activity model was adopted within two days (ignore-instructions—24-wonderful-road-to-100m).
- Frontier models pass POCs easily but often fail production on unit economics and latency: re-touching a whole retail catalog for Christmas with a giant model is “a 5-kilo hammer for adding snowflakes”. Smaller, controllable models win such jobs (hidden-layers—bria-misha-feinstein, misha-feinstein).
- Match architecture to the problem: encoders for extraction (business-context NER), fine-tuned open decoders where an explanation is needed (risk scoring people can trust, since a bare “88” means nothing), and frontier models mainly for prototyping and ground truth (hidden-layers—cyera-shiran-bareli, shiran-bareli).
- Profile before optimizing: most of Nexar’s 2.5 s latency was CPU preprocessing, not inference, and moving it to the GPU took hours rather than the planned quarter (hidden-layers—nexar-roni-goldshmidt, roni-goldshmidt).
- More context or more thinking can hurt. In deliberate workflows (not free agents), lower the reasoning budget on some nodes to cut variance (hidden-layers—qodo-itamar-friedman, itamar-friedman).
Disagreements & open questions
- Portability: Dor Cohen urges avoiding vendor lock-in with tested fallbacks (ai-engineering-podcast—ai-infra-at-scale); Wonderful builds every component model-agnostic and swaps per component (ignore-instructions—24-wonderful-road-to-100m); Gavriel Cohen argues agents can’t simply swap models (langtalks—71-claw-architectures).
- Can you hop between harnesses freely? A LangTalks host finds switching between Cursor, Claude Code and Codex nearly free with a harness-agnostic spec/plan/implement flow (langtalks—70-our-claude-code-tips); Gavriel Cohen follows Anthropic’s view that agents aren’t portable across providers and built no model-switching (langtalks—71-claw-architectures, gavriel-cohen).
Takeaways
- Before upgrading a model, narrow the task and rephrase the question; retest on the small model (ai-engineering-podcast—why-your-llm-costs-so-much)
- Split complex tasks so each small-model call has ≤2–3 logic steps (ai-engineering-podcast—why-your-llm-costs-so-much)
- Set reasoning-token budgets instead of default effort levels where available (ai-engineering-podcast—why-your-llm-costs-so-much)
Related
context-engineering · llm-inference · open-weight-models · llm-evals · ai-gateway · llm-pipelines · small-language-models · llm-cost-optimization