Guardrails

Also called AI guardrails, LLM guardrails

Quick answer

Guardrails are checks placed around an AI model that decide what may go in and what may come out. They block, change or flag requests and answers that break a set of rules, such as leaking personal data, obeying a hidden instruction or giving advice the owner does not allow. They run as ordinary software, outside the model.

~20 sec read

011 min

The problem guardrails solve

A traveller was planning to fly to a family funeral. Before booking, he asked the airline's website chatbot about reduced bereavement fares. Air Canada's chatbot told a customer, Jake Moffatt, that he could ask for a reduced bereavement fare after he had travelled.

A page on the same website said the opposite: a reduced fare was not allowed after travel. The airline refused the refund, and the customer took the dispute to a tribunal.

Nothing in the system had checked the chatbot's answer before the customer read it. The language model wrote a fluent sentence, and the sentence went straight to him. A British Columbia tribunal rejected the airline's argument that the chatbot was a separate party, and held that the chatbot was part of the airline's website, so the airline was responsible for what it said.

The missing step was a check on the answer before it left the system. That check is what the rest of this article describes, and it is easier to see once you follow one message through it.

chatbotfare after travelyespolicy pagefare after travelno
Look at the two answers to the same question. The chatbot and the policy page disagree, and nothing in the system compared them.

023 min

How guardrails work

Say a traveller asks an airline's support assistant whether a reduced fare can be requested after the flight. Without guardrails, the message goes straight to the language model and the model's answer goes straight back.

With guardrails, two checks are added to that path. The input check reads the message first and either passes it on or stops it. It looks for things the airline does not want the model to see, such as a credit card number, an abusive request, or instructions hidden in the message, a trick called prompt injection. The model then writes a draft answer. The output check reads that draft before anyone else does. If the draft answer disagrees with the policy, the output check blocks it and the traveller sees a fallback message instead.

Each check ends in a verdict. It can allow the text, block it, rewrite it, or flag it for a person to review while still letting it through. The fallback message is usually a plain sentence such as "I can't confirm that, here is the policy page."

The output check is also where a hallucination gets caught. A hallucination is an answer the model made up. Here the check compares the claim in the draft with the airline's real policy text, and it stops the answer when the two disagree.

Three ways to build a check

A guardrail check is one of three kinds, and many systems use all three.

Kind of checkHow it decidesStrong atWeak at
RuleMatches a word list or a pattern, such as a phone-number formatFast, cheap, easy to explainMisses any rewording or spacing trick
ClassifierA small trained model labels the text, for example "personal data" or "toxic"Handles wording the rules never listedNeeds training data and tuning
Language-model judgeA second model reads the text and answers a policy questionUnderstands meaning and contextSlowest and most costly

A 2023 paper introduced Llama Guard, a language model built to act as a safeguard on both the input and the output of a conversation. It sorts prompts and responses into the categories of a written list of safety risks, and its owners can change the list for their own use. That is the third kind in the table.

Rules that live outside the model

A 2023 toolkit called NeMo Guardrails lets developers add programmable rules to an application at runtime, apart from whatever the model learned in training. The developer writes them, can read them, and can change them without training the model again. That separation is the main reason guardrails are practical. The model stays general, and each product adds its own limits on top.

A few ready-made options exist. Open-source tools include NeMo Guardrails, the Guardrails AI framework, Llama Guard, and Presidio for finding personal data. Cloud services such as Microsoft Foundry and Cloudflare AI Gateway offer the same checks as settings, so a team can start without building them.

input checkpass or blockmodeldraft answeroutput checkpass or blockfallback message
Follow the message from left to right. The answer reaches the traveller only after the output check, and a blocked answer is replaced by a fallback message.

032 min

Guardrails for AI agents

A support assistant that only answers questions can do harm with words. An assistant that also acts can do harm with actions. An AI agent can look up a booking, issue a refund or send an email, and a wrong action is a refund already paid.

So guardrails for agents check more than two points. In Microsoft Foundry a guardrail is a named set of controls, and each control names a risk, the points to scan for it, and the action to take. Microsoft Foundry scans four points: the user's input, the tool call the agent proposes, the answer a tool sends back, and the final output. The tool call is the point that matters most for money and data, because the check can refuse the refund before it is sent.

The tool answer matters too. If the agent reads a web page or a document, that text can carry hidden instructions. A guardrail on tool responses treats everything the agent retrieves as untrusted.

Beside the agent or before it

Input checks cost time, so the order matters. An input check can run at the same time as the agent to save time, but if it then triggers, the agent may already have used tokens and run tools. The other choice is to run the check first and start the agent only if it passes. That costs a little more waiting but prevents any side effect from a request that should have been stopped.

The right choice depends on what the agent can do. For an agent that only drafts text, running the check beside it is usually fine. For an agent that can pay or delete, run the check first.

Checks that cover only part of a chain

Many agent systems hand work from one agent to another. In the OpenAI Agents SDK, input checks run only for the first agent in a chain, and output checks only for the agent that writes the final answer. A request that enters the chain through another route, or an answer written by a middle agent, is not covered by those two checks. Put tool checks on each action that matters.

The agent's guardrail replaces the model's

Settings can also hide a gap. In Microsoft Foundry an agent follows the guardrail assigned to it, and that guardrail fully replaces the one set on its underlying model. A team that tightens the model's settings but leaves the agent on a looser guardrail has not tightened anything for the agent. Check the guardrail that is actually assigned to the thing users talk to.

042 min

What guardrails check for

Teams do not start from a list of checks. They start from the failures they have seen or fear, and each failure maps to a check.

  • Harmful or abusive content. The check scores text for categories such as hate, violence and sexual content, and acts when the score passes a threshold the team sets. A lower threshold blocks more.
  • Prompt injection and jailbreaks. A jailbreak is a request written to make the model ignore its safety rules. OWASP's 2025 list of the top ten risks for LLM applications puts prompt injection first, as LLM01. Detectors look for the patterns of both.
  • Personal data. The check finds names, card numbers or health details in a message or an answer and masks or removes them before they reach the model or the user.
  • Off-topic requests. A bank's assistant should not give medical advice, and a check on topic keeps it to its job.
  • Unsupported claims. The check compares an answer with the source it should rest on, as in the fare example.
  • Format. If other software reads the answer, the check confirms it matches the expected structure, such as valid JSON, and asks for a retry if not.
  • Brand and compliance. The check blocks competitor names, promises the company cannot keep, or advice a regulator restricts.
  • Access. The agent's account holds only the permissions its job needs, so even a fooled agent cannot reach what it should not. This is least-privilege access.

Not every item is a text check. Some are settings on the systems around the model, and they belong on the list because a guardrail program is the whole set of controls, not one filter.

051 min

A test of 3,000 hours

How well do these checks hold when people try hard to break them? One published test gives a measured answer.

A 2025 research paper on constitutional classifiers reported more than 3,000 estimated hours of red teaming in which no tester found a universal jailbreak that pulled information from an early classifier-guarded model at the detail an unguarded model gave, across most target queries. A universal jailbreak is a single prompting method that works across many harmful questions. The result covered one early version of the system, and red teaming means paying people to attack a system on purpose. For more on that practice, see red teaming.

The authors also reported what the protection cost. The same classifiers raised refusals on production traffic by an absolute 0.38% and added 23.7% to inference cost. That is a small rise in wrongly refused questions and a noticeable rise in computing work. Both are normal for guardrails: every check adds some blocked good requests and some delay, and a team has to decide how much it will accept.

062 min

Where guardrails fail

A guardrail reduces risk. It does not remove it, and four limits come up in practice.

Attackers adapt

Detectors are models too, and they can be fooled. In a test of six detection systems, among them Azure Prompt Shield from Microsoft and Prompt Guard from Meta, character tricks and machine-learning attacks got past the detectors, in some cases 100% of the time. The attacks changed the characters of a message so a detector missed it while the target model still understood it. A detector that works on plain English can also miss attacks written in other languages.

Good requests get blocked

A check that is too strict stops legitimate users. A traveller asking about a medical emergency flight can look like a request for medical advice. The cost of a wrong block is a lost customer, so teams track the rate of blocked good requests as closely as the rate of caught bad ones.

Every check adds delay

A rule runs in a blink. A language-model judge takes a full model call. Teams place the slow checks only where the risk is high, or run them beside the model as described above, and accept the trade.

A pass is not proof of truth

A guardrail passes an answer that breaks none of its rules. That answer can still be wrong. The fare answer in the opening sounded polite, safe and on topic, and only a check against the policy would have caught it.

No single check covers all of this. Teams use several: rules for the obvious cases, a classifier for the rest, a judge for the hard ones, narrow permissions for actions, and a human-in-the-loop review the cases the checks flag. They also log every verdict, so they can see where the checks block too much or miss too much.

071 min

Guardrails vs system prompts and alignment

Guardrails are often mixed up with two nearby ideas, and the difference decides where to put your effort.

A system prompt is a set of instructions written to the model, such as "never discuss refunds". The model reads it with everything else and can be talked out of it. A guardrail runs as separate code and does not depend on the model agreeing. If a rule must always hold, put it in a guardrail and keep the system prompt for style and scope.

Training-time alignment

AI alignment is the work of making a model behave as its builders intend, mostly through training. A model trained to refuse is less likely to need a guardrail, but the training is fixed inside the model and the company that trained it controls it. Guardrails are the part an application team controls. A company that uses a hosted model cannot retrain it, but it can add checks in front of and behind it. The two work together: training makes the model's first answer better, and guardrails catch what is left.

082 min

How to put guardrails in place

Start from failures, not from tools. A short plan works better than a long one.

  1. List the five worst things the assistant could say or do, using real conversations if you have them.
  2. For each, choose the point where it can be caught: input, tool call, tool response or output.
  3. Start with the cheapest check that could work, usually a rule, and add a classifier or judge only where rules fail.
  4. Run the checks in log-only mode first. Read what they would have blocked, and tune the thresholds before they block anything.
  5. Switch to blocking for the severe cases. For an agent that can act, make the check fail closed: if it cannot decide, the action waits for a person.
  6. Log every verdict, review a sample each week, and attack your own system on a schedule. Rules go stale as attackers and products change.

Different roles own different parts. A PM decides which failures are unacceptable and how much blocked good traffic is tolerable. A designer writes the fallback message so a blocked user knows what to do next. An engineer places the checks, sets the latency budget and narrows the agent's permissions. Rules in some industries, such as finance and health, also set what must be logged and reviewed.

Before you ship, ask one question: if this assistant says something false with confidence, which check would stop it, and have you tried to make it fail?

?3 questions

Questions people ask

Can I use ready-made guardrails instead of building my own?

Yes. Open-source tools such as NeMo Guardrails, Guardrails AI and Llama Guard cover common checks, and cloud services offer them as settings. You still write the rules for your own product and test them.

Does passing a guardrail mean the answer is correct?

No. A guardrail only checks the rules it was given. An answer can break none of them and still be false, so factual claims need a check against a trusted source.

Who is responsible when a guardrail misses something?

The company that runs the assistant. In the Air Canada case, the tribunal held the airline responsible for its chatbot's answer, and it did not accept that the chatbot was a separate party.

§8 sources

Sources

Show all 8 sources
  1. Mondaq, a legal summary of Moffatt v. Air Canada, 2024 BCCRT 149:

  2. Rebedea et al., NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails, arXiv 2310.10501 (2023):

  3. Inan et al., Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations, arXiv 2312.06674 (2023):

  4. Sharma et al., Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming, arXiv 2501.18837 (2025):

  5. Hackett et al., Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems, arXiv 2504.11168 (2025):

  6. OpenAI Agents SDK documentation, Guardrails (vendor documentation, for that SDK only):

  7. Microsoft Learn, Guardrails and controls overview in Microsoft Foundry (vendor documentation, for that product only, preview features as of October 2026):

  8. OWASP Gen AI Security Project, LLM01:2025 Prompt Injection:

Keep reading

More from AI & Engineering

All of AI & Engineering
All of AI & Engineering

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.