liad-elidan (CEO of Milestone, engineering productivity) explains how to measure whether coding agents actually help. Pre-GenAI productivity relied on metadata (PR cycle, coding, and review time; DORA metrics), which ignored context and complexity. GenAI made the data itself readable: code lines, git blame, ticket content, and agent events from Copilot, Cursor, and Claude Code. That makes attribution possible: which commit or PR used which tool. Milestone’s framework has three layers. Adoption asks who uses which tools and features. Productivity compares cycle and review times with and without AI per developer. Quality measures code survival and longevity, tracing bugs back via git blame to whether the offending lines were AI-assisted. Hosts add: separate usage from utilization, check whether buggy code was spec-driven and used the internal MCP, and give custom agents success baselines that improve per version.
Show: langtalks
Concepts: ai-engineering-metrics · ai-sdlc · llm-evals · llm-cost-optimization