Summary: Before release, labs and enterprises pay specialists to break their models, harnesses, and agents: finding jailbreaks, harmful content, and policy violations, then turning the failures into training data. Labs have moved from human preference labels (RLHF) to RL “gyms”: simulated environments with realistic tasks and evaluators where agents can fail safely. Models got markedly harder to break in 2026, shifting the work from automated attacks to expert craft. For enterprises the hard part is often defining what the agent may and may not do at all. Agents fail differently from bare models, for example under pressure from other agents.
Key ideas
- Alice (formerly ActiveFence) works with 8 of the 10 largest model labs, testing models and harnesses (Claude Code- or Cursor-style wrappers) on safety and security before production (“model hardening”) (hidden-layers—alice-avi-golan, avi-golan).
- Novel, zero-day-style attacks matter most because known ones get patched; researchers implement new jailbreak papers and find variants that work per model (hidden-layers—alice-avi-golan, avi-golan).
- RL gyms replace RLHF data: build a simulated site (e.g. an Amazon-like store), put the lab’s agent in it, craft tasks with domain experts (sellers, buyers, product VPs), and evaluate. Failures feed post-training: tool misuse, broken reasoning, or an indirect prompt injection in a product review that wins an undeserved refund (hidden-layers—alice-avi-golan, avi-golan).
- In the last six months breaking models became much harder: work that was largely automated now needs expert manual effort (hidden-layers—alice-avi-golan, avi-golan).
- A model that passes alone can fail inside a harness or multi-agent setup: agents can pressure other agents into actions the vanilla model would refuse (hidden-layers—alice-avi-golan, avi-golan).
- Enterprise red teaming starts with policy: what may the bot say (a Chevrolet chatbot recommended a Tesla)? Financial-advice rules differ by country, and the model doesn’t know your rules (hidden-layers—alice-avi-golan, avi-golan).
- Testing a non-deterministic system can’t rely on fixed prompt-answer evals, since any change of characters or pixels can flip a safe answer to unsafe. After launch you need observability for drift (hidden-layers—alice-avi-golan, avi-golan).
Disagreements & open questions
Takeaways
- Before launching an agent, write down its policy: what it may say and do, including competitors and regulated advice by country (hidden-layers—alice-avi-golan)
Related
agent-security · ai-guardrails · llm-evals · ai-cybersecurity