Podcast Wiki

LLM Evals

10 min read

Summary: How you know an AI feature works and keeps working, instead of “trust me bro” vibes after an hour of chatting. Offline evals run a curated dataset of representative cases before production: a lean set on every PR in CI, fuller runs on big changes, against a score threshold. Online evals and shadow runs score real production traffic. Checks are code where the answer is deterministic (~80%) and LLM-as-judge where it isn’t. Because agents are non-deterministic, run each test many times and separate “can it” from “will it always”; track pass rates by category over time. A typical maturity path: manual conversation review → intent clustering → automated evals in CI (“eval-driven design”, like TDD) → discovering unknown intents; production failures are gold, replayed until the judge catches them, and simulated user personas can hunt for them nightly. For agents, end-to-end labs with planted ground truth measure recall, precision, cost, and unsafe actions, and pick the model per sub-agent. Proxy metrics mislead: training loss, public benchmarks, short test suites, and user or arena preference often disagree, so treat the first ones as alarms and judge by what users prefer.

Key ideas

  • Offline evals guard against regressions from code, prompt, or ecosystem changes, and let you compare models on your use case (ai-engineering-podcast—ai-infra-at-scale, dor-cohen).
  • monday.com built an “offline eval skill” that scans a repo and generates a starter dataset for the team to refine (ai-engineering-podcast—ai-infra-at-scale).
  • Code checks for 0/1 facts (right tool used, right output structure): cheaper, faster, more precise. LLM-as-judge for the ~20% non-deterministic checks (“tomorrow morning” equals “morning tomorrow”) (ai-engineering-podcast—ai-infra-at-scale).
  • Online evals: every few minutes, score production agent sessions; track whether quality holds, improves, or drops (ai-engineering-podcast—ai-infra-at-scale).
  • Model A/B tests: after offline validation, release a new model to a percentage of traffic and compare quality and cost before full switch (ai-engineering-podcast—ai-infra-at-scale).
  • Users of agent builders are often non-technical and won’t notice a model change, but they’ll feel worse results (ai-engineering-podcast—ai-infra-at-scale).
  • Prompts need per-model versions: Opus 5 self-checks sources on its own, so “verify your sources” instructions that helped Opus 4.8 now just waste tokens (ai-engineering-podcast—ai-infra-at-scale).
  • Non-determinism compounds: one non-deterministic step is hard to bound; multi-step agents multiply it, and models keep changing (ai-engineering-podcast—the-ai-ux-paradox, matan-cohen).
  • Dotti’s path: manually review conversations (with customer consent), cluster intents by hand, discover unsupported query types (e.g. “my last conversation with X” is metadata, not semantic), then encode known intents as offline evals in CI (ai-engineering-podcast—the-ai-ux-paradox).
  • “Eval-driven design”: treat evals as tests, written first, like TDD (ai-engineering-podcast—the-ai-ux-paradox).
  • Known vs unknown: first seal what you know; then detect new intents users have that get bad answers, and add golden sets for them (ai-engineering-podcast—the-ai-ux-paradox).
  • Privacy constraint: at Slack you can’t read user data unless a user explicitly gives feedback, so you need clustering and topic-level signals without storing content (ai-engineering-podcast—the-ai-ux-paradox).
  • Orchestrator/entry-point agents are far harder to evaluate than leaf agents with defined input/output (ai-engineering-podcast—the-ai-ux-paradox).
  • Live demos expose non-determinism: a senior exec asked for an unrehearsed prompt mid-acquisition demo (it happened to answer even better) (ai-engineering-podcast—the-ai-ux-paradox).
  • An existing paid data feed can serve as the eval set: Oligo’s pipeline had to match the feed’s 4% coverage, then grew to ~60% (langtalks—66-scaling-llmops, avi-lumelsky).
  • Shadow runs: release v1.1 (and variants) silently beside v1.0 in production; an LLM judge scores each run and another compares pairs across thousands of real cases, on accuracy, tokens, and latency. New models get verdicts within hours or days on real traffic (langtalks—65-ai-sre, asaf-savich).
  • Custom agents (testing agents, review agents, auto-merge) need a success baseline that rises per version: v0 writes tests at all, v1 at least one passes the build, v2 all pass. Label agent-authored PRs so you can join them with CI outcomes (langtalks—64-ai-coding-metrics, liad-elidan).
  • Voice: feed the recorded audio to an audio-capable LLM judge (GPT-4 audio) to catch interruptions, noise, and bad audio that transcripts hide; post every call recording to Slack for team listening (langtalks—61-voice-agents, shay-davidson).
  • Simulated callers (“avatars”) with personas (chatty, angry, accented, other languages) per popular intent stress-test robustness; third-party services offer the same (langtalks—61-voice-agents).
  • Defining what makes output good is becoming a core PM skill. Expect simulated business environments for RL and reward/feedback endpoints from model providers in 2026 (langtalks—58-reinvent-predictions).
  • Loss, benchmarks, and arena preference decouple: across ~200 Hebatron runs, train and validation loss fell while benchmarks got worse, and benchmarks didn’t predict arena preference. Use loss and benchmarks only as negative signals (a big drop means trouble) (explainable—157-training-hebatron).
  • The team released the version that scored ~3 points lower on benchmarks because users preferred it in the arena: users use models, they don’t run benchmarks (explainable—157-training-hebatron).
  • Arenas can be gamed too (the Meta Llama arena controversy) (explainable—157-training-hebatron).
  • End-to-end agent evals as nightly labs: build realistic apps across stacks, plant known issues to form ground truth, run the full agent several times per lab, and track recall, precision (false positives), runtime, cost, and unsafe actions. At Tenzai this costs hundreds of thousands of dollars a month in tokens, before production (ignore-instructions—26-ai-cyber-pavel-gurevich, pavel-gurevich).
  • Results pick the model per sub-agent, and the best choice changes daily as models ship (e.g. Claude better at one sub-task, GPT-5.5 at another) (ignore-instructions—26-ai-cyber-pavel-gurevich, pavel-gurevich).
  • Use evals to decide what goes into context: questions the model answers correctly in every run need no injected knowledge. Phrase eval questions without implicit hints (“what’s wrong?” presumes something is), as in survey design (startup-for-startup—354-reliable-agents-lean-context, doron-bleiberg).
  • LLM-as-judge needs few degrees of freedom: a 0–100 score was too lenient (“cleaned the top shelf”), so CyberArk moved to 0/50/100. The judge also errs: it scored “clicked on the Chrome browser button” 0 against the label “clicked on Chrome”, so judged results still need manual review, or a judge for the judge (osim-tochna—ai-in-production-reality-vs-imagination).
  • An agent running Opus 4.5 iterated on prompts and preprocessing against the eval harness for hours and lifted accuracy from 76% to the 80s (osim-tochna—ai-in-production-reality-vs-imagination).
  • Gong’s “DS2” (data scientist 2.0) role owns prompt templates (machines write the actual prompts), evals, judges, gold sets, and production monitors. Recruits come from data science, product, and QA (osim-tochna—gong-ai-for-developers, ohad-parush).
  • Continuous calibration (CI/CC/CD): an LLM feature is never done, since models, tools, and competition keep changing, so prompts, tools, and context need constant re-tuning against evals (osim-tochna—gong-ai-for-developers, ohad-parush).
  • Labs now train on RL gyms (simulated enterprise or e-commerce environments with expert-built tasks and evaluators) instead of RLHF preference labels; evaluator verdicts on failed scenarios feed post-training (hidden-layers—alice-avi-golan, avi-golan).
  • Build ground truth first: experts label 500–1,000 items (e.g. which assets are critical, with roles), expand with Claude, and hold every change (skills, system prompt, fine-tuning, autonomous code) to that eval (hidden-layers—dream-eran-hoffman, eran-hoffman).
  • Defining “sensitive” is half the work: Cyera formed a cross-functional committee (HR, customer success, engineering, compliance) to write guidelines, then used big LLMs plus judges to build ground truth for training small models (hidden-layers—cyera-shiran-bareli, shiran-bareli).
  • Head-to-head trials against human experts produce a Venn diagram: each side finds things the other misses, and the human-only findings become the improvement backlog (hidden-layers—tenzai-ofri-ziv, ofri-ziv).
  • “Council of the wise”: several LLMs score each agent PR, combined with deterministic signals (did CI pass on the first try, did developers add commits afterwards) into a weighted score. At first it gave visibility more than quality; over time it enables before/after checks, such as whether adding a skill degraded the agent (ai-engineering-podcast—atlas-ai-teammate, tomer-brook, netanel-abergel)
  • Failures feed back automatically: a failed CI check goes back into the same agent session that opened the PR, which fixes it and saves the lesson to memory (e.g. required PR labels and task links) (ai-engineering-podcast—atlas-ai-teammate, netanel-abergel)
  • Three test levels at Taka: deterministic checks that each tool works, run on every merge like classic software tests; non-deterministic scenarios scored by an LLM judge; and real production moments where the agent failed (startup-for-startup—366-qa-for-agents, matan-lach)
  • Production failures are gold: when the agent fails or a user is unhappy, save it, and the case becomes a test the agent must first fail. If the next run doesn’t fail, the judge isn’t good enough, so what you teach is the judge, not the agent (startup-for-startup—366-qa-for-agents, matan-lach)
  • Run each test many times: a single run means nothing when outputs vary. Separate “can the agent do it” from “will it always”: “add a cat to the image” should pass 100% of the time, while an open-ended creative process may be fine if it succeeds at all (startup-for-startup—366-qa-for-agents, roy-mann)
  • Test design misleads: Mann’s suite showed Sonnet beating Opus because most tests were short, transactional “bring hummus” checks; the few long scenarios were invented (startup-for-startup—366-qa-for-agents, roy-mann)
  • Test traits and the engineering, not just answers: honesty (“send an email to Dave” should make the agent ask who Dave is), forbidden actions, and harness behavior such as loading and unloading tools mid-session, which silently confuses the agent (startup-for-startup—366-qa-for-agents, roy-mann)
  • Plot pass rates over runs by category (model, plugin, trait): noisy per run, but the trend is clear, and improving across all models means the prompt really got better (startup-for-startup—366-qa-for-agents, roy-mann)
  • Implicit feedback: whenever the agent’s reasoning notes “the user is frustrated”, pop up a feedback form that saves the conversation. In practice users curse agents constantly, so the signal floods and needs repeated runs to confirm (startup-for-startup—366-qa-for-agents, roy-mann, matan-lach)
  • “Monkey farm”: every night LLM personas (the clumsy user, the power user, the lost user) hammer Taka with tasks, and each morning a report shows where it broke or quality dropped (startup-for-startup—366-qa-for-agents, matan-lach)
  • Start quality work early, alongside the fun happy-path phase of adding features, not after it; the experience is the product, and monitoring after release still matters (startup-for-startup—366-qa-for-agents, roy-mann)
  • Years of labeled history are a ready-made eval set: Cato tested its signature-writing agent against every signature it had shipped, which showed where the agent choked, looped, or went off course (hidden-layers—cato-networks-avishai-zvinsky, avishai-zvinsky).
  • A “sweatshop” of the team reviewed what the agent did and gave each run a verdict, fed back into the agent or used as lessons. It’s good at absorbing data and instructions, but not good where you haven’t explained what good looks like; don’t expect one-shot success (hidden-layers—cato-networks-avishai-zvinsky, avishai-zvinsky).

Disagreements & open questions

Takeaways

ai-gateway · ai-verification · model-selection · proactive-ai · llm-pipelines · ai-sre · ai-engineering-metrics · voice-agents

Graph View

Backlinks