sharon-dahan, an architect at the inference company Impala, explains what happens behind a chat-completion call: model size × quantization = memory; tensor and data parallelism; mixture-of-experts; the KV cache; and why idle GPUs, like idle taxis, lose money. Generic API pricing carries big margins, so latency-insensitive workloads can run for a fraction of the price. On model choice: size isn’t everything. Narrow the task, reframe the question, and manage context aggressively. A ship-ERP agent went from 85% to 97% on evals through context management alone. Also covered: Chinese open-weight models (open weights aren’t code), agents that interrupted each other and blocked a workday, an agent deleting a DB, and why to ignore any AI hype that hasn’t survived a month.
Show: ai-engineering-podcast
Concepts: llm-inference · model-selection · open-weight-models · context-engineering · ai-guardrails · ai-hype