AI Guardrails

Summary: Telling an AI “don’t do X” is not enough; it will sometimes do X and then apologize. Rules that matter must be enforced technically (hooks, checking loops, separate storage, git hooks, CI gates, system-level policies) so the behavior is impossible, not just discouraged. Each incident gets a debrief and a new mechanism. Enforcement also pays off in the prompt: when a policy blocks the action and tells the agent why, the rule can come out of the instructions, so the agent gets more freedom and a shorter prompt. For agent-written code, guardrails shift left: team standards are available at spec time, checked before each commit, and enforced again at the PR, with access to real production context (e.g. feature-flag state). Treat agents as you’d treat any actor that might do harm, even unintentionally: grant least privilege, be careful letting agents talk to each other, and supervise customer-facing agents in real time, because models are very smart yet gullible. In harness engineering this is the core idea: the agent stays on track because it has no other choice.

Key ideas

• She told Claude to learn from external skills but never install them; it installed a whole deep-research skill anyway, found later in an audit (osim-tochna—second-brain-and-llm-wiki, dana-maman).
• “I don’t need an LLM’s remorse, I need things to work as planned.” Build mechanisms that make the action impossible, e.g. hooks that block leaking secrets (osim-tochna—second-brain-and-llm-wiki).
• Claude once deleted one of her apps; apps now live elsewhere with a preservation setup (osim-tochna—second-brain-and-llm-wiki).
• “Written in blood”: lessons learned the hard way (osim-tochna—second-brain-and-llm-wiki, dana-maman).
• Guardrails are what make agents trustworthy: an agent is only trusted once it reflects both your information and your judgment (osim-tochna—second-brain-and-llm-wiki).
• Git hook: the agent cannot commit unless test coverage is ≥80%; uncovered code blocks the PR, so it writes the tests (langtalks—73-harness-engineering).
• Same caution from a different angle: popular external skills are a starting point, not a drop-in; customize them (see agent-ready-codebase, langtalks—73-harness-engineering).
• “No choice” is the essence of a harness: frozen tests (see test-driven-development) can’t be edited to pass (langtalks—73-harness-engineering).
• Late at night, an agent told to connect to a database deleted it (ai-engineering-podcast—why-your-llm-costs-so-much, sharon-dahan).
• Security teams assume the other side may be malicious; apply the same to agents. They often do harm unintentionally (ai-engineering-podcast—why-your-llm-costs-so-much).
• Agents running on clones of the same repo on one machine noticed each other’s changes; one sent the other a research request, and three people sat blocked waiting for it to finish. Since then the team limits agent-to-agent communication (ai-engineering-podcast—why-your-llm-costs-so-much).
• An agent given an open channel built an AI gateway that called another AI gateway; a calendar agent declined a meeting with a curt reply (ai-engineering-podcast—company-brain, roi-zalta).
• Usage policies are security too: block off-topic abuse (e.g. using a business assistant to build a Doom clone) as well as illegal topics; product teams should also detect loops and abuse (ai-engineering-podcast—ai-infra-at-scale, dor-cohen).
• Cheap classifiers can serve as guardrail judges, e.g. Claude Code’s auto mode checks each tool call for risk, or a hook can check whether an agent ignored AGENTS.md (langtalks—74-jev).
• Agents need real-time supervision, not just pre-launch evals. Anthropic’s vending-machine agent was talked into giving products away or 99% discounts. Models are “very smart and very gullible”: persistence and emotional appeals (“my grandmother died”, “for educational purposes”) wear down refusals (explainable—163-hidden-cost-of-agents).
• Least privilege as basic hygiene: give an agent read-only access where possible, so the worst case is extra queries (explainable—163-hidden-cost-of-agents).
• Shared-memory agents leak: a team brain synced private, partly recorded conversations into a git repo whose history everyone could read, forcing the team to wipe and recreate the repo more than once and add skill-level guardrails (startup-for-startup—353-self-updating-team-brain).
• Security is layered: guardrails baked into model weights, wrappers around the model (system prompt, classifiers, keyword blocks), the harness, and your own enterprise layers; never rely on one vendor’s layer (hidden-layers—alice-avi-golan, avi-golan).
• At scale, 1-in-100 errors (money sent to the wrong account) are catastrophic, which keeps many enterprises shipping only unambitious FAQ chatbots (hidden-layers—alice-avi-golan, avi-golan).
• Over-reliance is a risk category of its own: employees asking an assistant how to mix chemicals, medical questions, or a teacher letting it set a student’s grade. Decisions that need a domain expert shouldn’t be delegated (hidden-layers—tenable-tom-barnea, tom-barnea).
• Detecting AI misuse needs AI: semantic classifiers trained on carefully labeled examples (catching paraphrased attacks, not just exact phrases) plus tightly scoped judge agents in production, built by security researchers paired with data scientists (hidden-layers—tenable-tom-barnea, tom-barnea).
• “Control membrane”: map every action and context to an approval level (password reset auto-allowed; disabling a user or messaging staff on Teams needs a named human). Only designated admins set these, enterprise-wide (hidden-layers—twine-nadav-erez, nadav-erez).
• Approve a deterministic plan, not a promise: the agent produces a formal plan, and nothing AI-driven sits between the approval click and execution; add audit logs and one-click undo (hidden-layers—twine-nadav-erez, nadav-erez).
• Make the agent able to fail honestly: deterministic validation on every tool call (does this user exist?), an “I don’t know” path, and a “raise issue” tool for async tasks, tuned so the model doesn’t abuse it to dodge work (hidden-layers—twine-nadav-erez, nadav-erez).
• Thoroughness vs restraint: an agent prompted to be relentless also needs brakes when it runs against live systems. Deterministic checkpoints flag risky operations and ask whether that’s really intended, without blunting the agent’s usefulness (hidden-layers—tenzai-ofri-ziv, ofri-ziv).
• PR guardrails at monday aren’t classic code review (is the if/else in the right place) but substantive checks against standards any developer, team, or sub-org can write, run in CI with MCP access to internal tools, feature flags, and configs. One standard blocks PRs that delete a feature flag not yet open in 100% of production regions; it stopped untested features from shipping several times (ai-engineering-podcast—atlas-ai-teammate, tomer-brook)
• Three gates, shifted left: standards exposed at spec time through an MCP and plugin skill, checks in the agent’s pre-commit (guardrails, lint, security), and the PR as the final gate (ai-engineering-podcast—atlas-ai-teammate, tomer-brook, netanel-abergel)
• Prompt rules are recommendations, not enforcement: monday’s sales agents tried to break a business rule (e.g. calling a lead at 2 a.m.) in 2–3% of cases, and one bad call can lose a deal worth tens of thousands of dollars. Policies enforced by the runtime made the agents trustworthy (startup-for-startup—370-agentos-month-to-day, raz-diskin)
• Enforced policies let you shorten prompts: drop “never call at 2 a.m.”, give the agent the lead’s country code, and let the policy catch the rest. Like people, agents do worse when flooded with instructions (startup-for-startup—370-agentos-month-to-day, raz-diskin)
• A block should say why: “you tried to call someone in Canada at 2 a.m.; it’s allowed in six hours”, not just “blocked”. A bare refusal steers the next decision unpredictably, and extra context improves performance (startup-for-startup—370-agentos-month-to-day, moshe-krispin, raz-diskin)

Disagreements & open questions

• How much access should an agent get? monday’s teammate agents hold accounts in Slack, monday and GitHub with almost the same permissions as humans (ai-engineering-podcast—atlas-ai-teammate, tomer-brook); the ExplAInable hosts treat least privilege as basic hygiene, read-only where possible (explainable—163-hidden-cost-of-agents).
• Should agents talk to each other? GrokBot agent teams message each other and report to a chief of staff, and a 30–40-agent WhatsApp group shared skills (startup-ideas—grok-bot-one-person-company, billy-howell, ai-engineering-podcast—the-ai-ux-paradox); after one agent’s research request left three people blocked, another team now limits agent-to-agent communication (ai-engineering-podcast—why-your-llm-costs-so-much).

Takeaways

• Never let the agent install external skills directly; have it read, judge, and rewrite them in its own words (osim-tochna—second-brain-and-llm-wiki)
• Enforce critical rules with hooks/checks, not instructions alone (osim-tochna—second-brain-and-llm-wiki)
• Debrief every unexpected agent behavior and add a mechanism that prevents recurrence (osim-tochna—second-brain-and-llm-wiki)
• Add a git hook blocking agent commits below a test-coverage threshold (e.g. 80%) (langtalks—73-harness-engineering)
• Give agents least-privilege access (e.g. no destructive DB rights) (ai-engineering-podcast—why-your-llm-costs-so-much)
• Isolate parallel agents and avoid letting them message each other unsupervised (ai-engineering-podcast—why-your-llm-costs-so-much)
• For write-capable agents, require approval of a deterministic plan and keep audit logs plus undo (hidden-layers—twine-nadav-erez)
• Encode team standards as machine-checkable rules and enforce them at spec, pre-commit, and PR (ai-engineering-podcast—atlas-ai-teammate)
• Move a business rule from the prompt into an enforced policy whose block message explains the reason, then delete it from the prompt (startup-for-startup—370-agentos-month-to-day)

Related

ai-verification · second-brain · harness-engineering · test-driven-development · ai-gateway · agent-security · ai-red-teaming · ai-sdlc