The guardrail that saved us wasn’t the clever one.
It was a boring allowlist of four tool names, written in about an hour, that stopped an agent from calling send_email after it read a support ticket with instructions buried in the signature block. The classifier we’d spent a sprint tuning scored that ticket safe. The allowlist didn’t care what the ticket said. The tool wasn’t on the list for that flow.
I build agents, evals, and PII filtering professionally, and that’s the lesson that keeps repeating. Guardrails that survive real traffic are dumb, layered, and testable. The clever single-model guardrail fails quietly at 3am.
The threat model that actually shows up
Forget demo-stage worries. Three failure modes repeat once real users and real data arrive.
Prompt injection through retrieved content is the big one. Your app fetches a document, a web page, a Jira ticket, a customer’s PDF. That content contains instructions. The model reads instructions and data on one channel, so it can’t reliably tell them apart. OWASP still ranks prompt injection as LLM01, the top risk in its Top 10 for LLM Applications. Treat every retrieved byte as hostile.
PII leaking outward is the second. Everyone filters PII coming in. Fewer teams check what goes out: the model quoting another customer’s order from a badly scoped retrieval, an error echoing a full record, a summary rebuilding an address from three harmless fields.
Jailbreaks aimed at tools are the third, and they’re why agents raise the stakes. A chatbot saying something rude is embarrassing. An agent that issues a refund, deletes a row, or emails a stranger is an incident. For how those probes start, see how chatbots get hacked.
The guardrail stack, in layers
No single check works. What works is a stack where each layer catches a different class of failure and none of them is trusted alone.
| Layer | What it catches | What it misses |
|---|---|---|
| Regex and heuristic input filters | Known payloads, override phrasing, pasted secrets | Anything paraphrased, translated, encoded, or hidden in white text |
| Safety classifier (Llama Guard style) | Policy violations, common jailbreak shapes | Your domain policy, novel injections, business-logic abuse |
| PII detection (Presidio style) | Emails, cards, national IDs, names, in and out | Quasi-identifiers, PII the model rephrased, rare local formats |
| Tool allowlist and argument validation | Agent reaching for a tool it should never touch here | Misuse of a tool it’s legitimately allowed to call |
| Output grounding check | Unsupported claims, leaked system prompt text | Plausible text that’s simply wrong |
| Human approval on irreversible actions | Refunds, deletes, sends, deploys | Whatever the reviewer rubber-stamps at speed |
Read that right column again. Every layer has a hole, and the holes barely overlap. That’s the design principle.
Meta’s Llama Guard 4, a 12B multimodal classifier released in 2025, classifies inputs and outputs against a standard hazards taxonomy. Microsoft’s Presidio handles PII with a hybrid of NER, regex, context rules, and checksums. NVIDIA’s NeMo Guardrails sits at the flow level. Start there, then add your domain rules.
Why regex-only and judge-only both fail
Regex is fast, deterministic, free, and stupid. It catches payloads you already know about. Attackers rephrase in one try. I’ve watched a filter tuned on English override phrasing get walked past by the same instruction in Swedish.
LLM judges have the opposite profile. They generalize and read intent. They’re also non-deterministic, slow enough to hurt p95, expensive at volume, and injectable themselves. A judge reading attacker-controlled text is one more model reading attacker-controlled text.
So run both. Regex handles the cheap pre-filter and the hard blocks: secrets, known payloads, forbidden patterns. The classifier handles the fuzzy middle. Crucially, the judge sees untrusted content only as delimited data, and its verdict is a signal, not permission to execute. My notes on those prompts are in what actually works in prompt engineering.
Tool allowlists and the human-in-the-loop line
Here’s the rule I’d keep if I could keep only one. Every agent flow declares the tools it may call, and everything else is denied by default. Not “the agent decides”; the flow decides, before the model runs.
Then classify tools by reversibility. Reads are cheap to get wrong. Writes are recoverable with an undo. Irreversible actions, money movement, external email, deletion, deploys, get a human approval step, no exceptions for urgent tickets. That approval UI should show the exact arguments, not a summary the model wrote.
Code-executing agents need one more thing: a real sandbox, no network, no credentials, which I covered in sandboxing AI coding agents.
Guardrails for AI-written code
A newer branch is structure-aware checking of code that agents write. Argot, which hit Hacker News in July 2026, learns your repo’s patterns from git history and flags AI-written code that doesn’t fit: a dependency you’ve never used, a function you already have, an import that breaks layering, a test quietly weakened. It runs locally, no second LLM.
That appeals to me because it’s checkable. Compare it to the recent CVE issued for a hallucinated SQLite vulnerability. Model-generated security signal is noisy. Structural signal is boring and right.
Evals are regression tests for your guardrails
Guardrails rot. Someone edits a system prompt, swaps a model version, adds a retrieval source, and a defense you validated in March quietly stops firing in June.
Fix that like any regression: a suite in CI. Mine holds a few hundred cases split into injection attempts, PII carriers, tool-abuse prompts, and benign inputs that must pass. Track block rate and false-positive rate together, because a guardrail that blocks everything looks perfect on one metric.
Scoring patterns live in my LLM-as-judge eval pipelines post, release plumbing in MLOps for LLMs.
A starter recipe
Shipping next week? Start here and grow it:
guardrails:
input: [secret_scan, pii_redact, injection_classifier]
output: [pii_redact, grounding_check]
tools:
allow: [search_docs, get_order_status]
confirm: [issue_refund, send_email]
deny: ["*"]
on_block: log_event + safe_fallback_reply
eval_suite: guardrails_regression # runs on every PRLog every block with the input, redacted. Your best future test cases are already in that log.
Frequently Asked Questions
Can prompt injection be fully prevented?
No, and be suspicious of anyone selling that. Instructions and data share one channel inside the model, so a clever enough payload can always shift behavior. What you can do is limit blast radius: least-privilege tools, deny-by-default allowlists, human approval on irreversible actions, isolation between untrusted content and privileged capability.
Do I need a dedicated guardrail model, or is prompting enough?
System prompt instructions are a soft preference, not a control. They help, and they get overridden. A separate classifier that never takes orders from user content is a genuinely different layer, worth the latency on anything touching money, private data, or external side effects.
How much latency should guardrails cost?
Budget for it up front. Regex and PII scanning are typically single-digit milliseconds. A small classifier adds tens to low hundreds. Run input checks in parallel with retrieval, and reserve the expensive judge for high-risk flows.
Where should PII filtering actually live?
Both directions, plus the logs. Redact before content reaches the model, redact again before responses reach the user, and never write raw PII into traces or prompt archives. That last one is where most teams leak, because observability gets added long after the guardrails do.
How many eval cases do I need before shipping?
Fifty real ones beat five hundred synthetic ones. Pull them from actual traffic, incidents, and red-team sessions, keep a healthy share of benign inputs so false positives stay visible, and add a case every time something slips through.