Host: Daria Wertheim. Part of the “time capsule” series on current thinking about agents.
roy-mann and matan-lach (who leads Taka, an AI social media manager from monday’s Agent Labs) discuss measuring agent quality when the same input never gives the same output. Agents range from “pass the butter” task bots to personal assistants that must remember you; Taka went vertical because general agents lacked social-media expertise. Testing has three levels: deterministic tool checks in CI, LLM-judged scenarios run many times, and production failures replayed until the judge catches them. Plus trait tests (honesty), a nightly “monkey farm” of LLM user personas, and frustration-triggered feedback. Pitfalls: Sonnet “beat” Opus on short transactional tests, and tool loading breaks agents silently. Pass-rate trends across models show real prompt gains. Advice: start quality work early. The experience is the product, and candor about what the agent couldn’t verify builds trust.
Show: startup-for-startup
Concepts: llm-evals · agent-architectures · context-engineering · ai-verification