Podcast Wiki

LLM Cost Optimization

7 min read

Summary: Managing LLM spend has two halves: measuring it and reducing it. Measuring (AI FinOps) treats AI like any cloud cost. Provider bills lump calls together, so tag every call at a gateway or internal proxy, log the telemetry, and compute full unit economics per use case: tokens plus observability, subscriptions, databases and infrastructure. Reducing comes down to engineering habits. Put stable content first so prompt caching works (cached tokens cost roughly a tenth). Output costs more than input, so ask for short, structured answers. Plan with a strong model and execute with a cheap one (or route each step to the cheapest model that can do it), and trim whole traces, not single prompts. Cost tends to become the main problem once agents reach scale, when a dollar or two per run is multiplied by thousands of runs. For coding agents, spend per developer is rising; the goal is value per dollar, not cutting to zero.

Key ideas

Measuring: attribution and unit economics (AI FinOps)

  • Bedrock’s cost report can’t tell which of many systems made a call. Inference profiles (logical model IDs per team) helped but don’t scale; the shift is to gateway tagging and call logs exported (e.g. to S3) (langtalks—67-finops-for-ai, yizhar-gilboa).
  • Simplest implementation: an internal proxy exposing the same model API but requiring metadata (repo, use case) on each call (langtalks—67-finops-for-ai).
  • Holistic unit economics: customer-support cost equals agent tokens plus its database, backend, and tooling, the same way you’d break down storage per GB or DB queries (langtalks—67-finops-for-ai).
  • Coding spend per developer: ~65% of companies 300–600, ~5% under $100 (langtalks—67-finops-for-ai).
  • It must not be zero: every developer can benefit; the best treat themselves as leads of small agent teams (langtalks—67-finops-for-ai).
  • Analyze the extremes: non-users need adoption help (education, guilds, peer review); heavy spenders’ sessions reveal infrastructure fixes (e.g. if code exploration eats 30% of tokens, maintain docs/indexes to cut it) and good practices to spread via skills (langtalks—67-finops-for-ai).
  • AI in production costs much more than AI for coding; companies still happily open the tap for coding when it boosts productivity (langtalks—67-finops-for-ai).
  • Prediction: the theoretical $1–2k/month per-developer ceiling will break; money becomes tokens, intelligence, energy (langtalks—67-finops-for-ai).
  • Mandatory project tags on an internal LLM API answer whether, say, the back office is half the Bedrock bill or negligible (langtalks—66-scaling-llmops).
  • Spend should be read alongside productivity metrics: productivity per dollar (langtalks—64-ai-coding-metrics).
  • Hidden production costs demos skip: calling a model on another cloud (e.g. Gemini from AWS infrastructure) adds data egress fees; EU customers may forbid data leaving Europe, so global routing for capacity overflow is off-limits; you must verify the vendor isn’t logging or retaining data (osim-tochna—ai-in-production-reality-vs-imagination).

Reducing: engineering habits

  • Caching works from the top down until the first change. Put the system prompt (generic, repeated) first and per-user and per-task data after (dor-cohen, ai-engineering-podcast—ai-infra-at-scale).
  • Anti-pattern: a timestamp at the start of every prompt broke the cache before it began (ai-engineering-podcast—ai-infra-at-scale).
  • In chat, every turn resends the whole history (N+1), so good caching of the prefix matters a lot (ai-engineering-podcast—ai-infra-at-scale).
  • Personalization context that rarely changes (who the user is, their boards) belongs in the stable prefix (ai-engineering-podcast—ai-infra-at-scale).
  • Output tokens cost more than input. Chatty, over-explaining answers cost you and the user, so constrain structure and length (ai-engineering-podcast—ai-infra-at-scale).
  • Plan vs execute: plan with Opus, execute with Sonnet, Haiku, or another model, in products as in your own dev workflow (ai-engineering-podcast—ai-infra-at-scale).
  • Cost arrives after scale: monday’s sales-agent team first fought scale (hundreds of thousands of concurrent events), then cost, at ~$1.50 per agent run across thousands of leads. The infra group is building a model router that sends each step to Opus, Sonnet or Haiku as needed (startup-for-startup—370-agentos-month-to-day, moshe-krispin).
  • Prompt routing mid-session breaks the model’s cache. A gateway-level cache can add consistency (ai-engineering-podcast—ai-infra-at-scale).
  • For pure classification at scale, a dedicated classifier can be ~30–100× cheaper and far faster than a small LLM with thinking (see decision-classifiers, langtalks—74-jev).
  • Optimize whole traces, not single prompts: fewer tool calls, and smaller tool responses that keep the needed context (strip headers, reformat JSON to YAML) (langtalks—67-finops-for-ai).
  • Prompt compression is back: a community repo reportedly raised GPT 5.2 quality ~30% without extra cost or latency; startups mine traces for cheaper call patterns and caching (langtalks—67-finops-for-ai).
  • Iterative real-time analysis (re-asking with all prior events each time) benefits hugely from Bedrock prompt caching, since most input repeats (langtalks—66-scaling-llmops).
  • Rate limits: cross-region inference profiles gave ~5× the throughput of a region-pinned model (e.g. EU Sonnet → global), while staying compliant; some companies also spread load across multiple accounts (langtalks—66-scaling-llmops).
  • Tool overload (100+ tools) hurts, but swapping tool subsets per step breaks the KV cache. Manus masks token logits to restrict tool choice while keeping the prefix stable (langtalks—55-context-engineering).
  • Agents are expensive at scale: the OpenClaw creator reportedly ran 100 Codex agents nonstop for a month for ~500k just to build a working multi-agent system (team, architecture, memory, RAG), before infra and tokens (explainable—163-hidden-cost-of-agents).
  • Subscriptions are subsidized: a heavy Claude Max 1.5–2k at API prices (~1:10). If vendors switch to AWS-style metered billing, economics change (explainable—163-hidden-cost-of-agents).
  • Every skill, hook, and plugin loaded into context costs tokens on each send; a big context window fills fast and burns the weekly quota (explainable—163-hidden-cost-of-agents).
  • Inference optimization is a growing role: minimize $/1M tokens while meeting the SLA, since every token costs GPU time and electricity (explainable—163-hidden-cost-of-agents).
  • Kubernetes FinOps beyond tokens: Riskified combined spot instances with reserved instances and savings plans, and built a controller that steers Karpenter’s spot/on-demand mix by how well the org’s commitments are being used. Almost no servers run at full price (all >50% off), saving 30–40% on Kubernetes. Installing a tool by its quick-start isn’t enough; value comes from understanding your environment (osim-tochna—running-llms-at-scale, kfir-schneider).
  • Margins differ by feature: agent features run close to raw LLM cost, especially when users can pick the priciest model, while non-AI automations keep software margins, so a product’s blended margin depends on its usage mix (startup-for-startup—369-ai-pricing, roy-mann)

Disagreements & open questions

Takeaways

ai-gateway · llm-inference · model-selection · harness-engineering · ai-engineering-metrics · decision-classifiers · ai-pricing

Graph View

Backlinks