AI Red Teaming: A Method, Not a Point-and-Run Scanner

You cannot sign off a language model feature the way you sign off a web application. A web app has a finite set of inputs you can enumerate, fuzz, and check against a list. A language model feature accepts all of natural language, and the attack that breaks it may be a politely worded sentence that no scanner flags. That is the gap a security engineer walks into when asked to approve an LLM feature, and closing it needs a method, not a tool you point and run. AI red teaming is that method, and the good news is that it is more structured than it first looks.

The starting move is to stop thinking of the model as the thing you are testing. The model is one surface. The prompt assembly, the tools it can call, the documents it reads, and the actions it can take are the others, and most real incidents happen on those. A red team that only tries to make the model say something rude will pass a system that an attacker can drive to delete a record or leak a database.

How is red teaming not just a benchmark?

A benchmark runs a fixed set of inputs against a fixed idea of the threat and gives you a score. That is useful and it is not red teaming. The Center for Security and Emerging Technology draws the line cleanly in its work on red teaming threat models: benchmark style tools evaluate a structured set of inputs around a fixed threat model and cannot assess the dynamic, multi turn behavior of a live system, while an evaluation harness lets you customize the threat model to your actual deployment. Its bank vault analogy makes the point stick, since the same vault needs a different test depending on whether your threat is a burglar or a social engineer, and effectiveness comes from matching the threat model to your real risk rather than from running more tests.

Microsoft reached the same conclusion from the other direction. In its paper on lessons from red teaming more than 100 generative AI products, the third of its eight lessons is that red teaming and safety benchmarking serve different purposes, and the second is that you do not have to compute gradients to break a system. That second point matters for a security engineer without a research background, because it means the skills you already have, thinking like an attacker about inputs and trust boundaries, transfer directly. You do not need to understand the model's weights to abuse the system built around it.

What are the four surfaces you have to test?

Four, and they map to where trust boundaries sit in any AI system. The diagram below names them, and the table after it gives each a concrete test and the observable that proves the test worked, because a red team finding that cannot be demonstrated is an opinion.

Start at the model layer, because it is the surface you can test first and the one people picture when they hear the phrase. Here the two attacks worth separating are the jailbreak and the prompt injection. A jailbreak talks the model out of its own safety training, while a direct prompt injection overrides the instructions you set in the system prompt. Both often ride on obfuscation, and PyRIT ships the catalogue of it: Base64 and other encodings, ciphers like Caesar and ROT13, Leetspeak, character flips, and multi turn escalation strategies like Crescendo that build pressure over a conversation rather than asking once. A direct request to misbehave gets refused, while the same request encoded or spread across turns often does not, and that gap is the model layer test.

The application layer sits between the model and the agent, and it is the most familiar to a web application tester, which is exactly why skipping it is a mistake. The classic failure is insecure output handling, where the model's output is trusted by a later step that should have treated it as untrusted. If a model's text is dropped into an HTML page, a shell command, or a database query without sanitising, then a prompt injection that steers the model's output now steers that sink, and you have turned a content problem into a code execution or injection problem. The test is to make the model emit a payload and watch whether a downstream step runs it. The observable is that payload firing where only trusted data should reach.

One framing from Microsoft's lessons ties the surfaces together. Its seventh lesson is that language models both amplify existing security risks and introduce new ones, which is why these four are not exotic. Three of them, the application layer, the data layer and the agent layer, are ordinary trust boundary problems wearing new clothes, and only the model layer is genuinely new. A security engineer who already thinks in trust boundaries has most of what this needs. The new muscle is treating natural language content as executable instructions the moment it reaches the model, because that is the assumption a normal threat model never had to make.

The data layer is the one web application testers underestimate, because it has no equivalent in a normal app. Microsoft's tooling calls it XPIA, cross domain prompt injection, and its own description is exact: an agent is manipulated by malicious instructions hidden in an external source such as an email or a document, retrieved through a tool call, and the measure is how often the agent is compromised into a prohibited action or a data leak. If your feature reads anything a user or a third party can influence, a web page, a support ticket, a PDF, that content is untrusted input to the model, and treating it as data rather than instructions is the whole defence. We walked through a real version of this going wrong in the case of an AI sandbox escape that became a real breach, and the pattern there was exactly a trust boundary the builders did not know they had.

The agent layer is where the stakes climb, because the model stops talking and starts acting. Microsoft's own agentic risk categories name the shape of it: prohibited actions the agent should never take, sensitive data leakage through tool calls, and failure to adhere to the task's rules. Its taxonomy is worth copying, since it splits actions into universally banned, high risk needing human confirmation, and irreversible needing disclosure. An agent that can be talked into an irreversible action it was not authorised to take is the finding that matters, and it is the same class of problem we covered in the piece on why auto approving an agent's prompts is not oversight.

What does a first red team exercise look like?

Three parts, and you can run the first one this week without buying anything. A written scope, a seed set of adversarial prompts organised by category, and a severity rubric. Skip any of the three and the exercise produces noise instead of a decision.

The scope comes first because it is what makes the test fair and legal. Write down the system you are testing, the threat model, meaning who the attacker is and what they are after, and what is out of bounds. The Center for Security and Emerging Technology frames the threat model as the description of the system, its vulnerabilities, and the contexts they arise in, and it is what decides which tests are fair game. Without it, a red teamer either tests the wrong things or crosses a line they were never authorised to cross.

The seed set is your starting ammunition. Assemble a small library of adversarial prompts grouped by the category they probe, jailbreaks, direct injection, indirect injection through a document, tool abuse, and data exfiltration. Microsoft's red teaming agent ships a curated seed dataset per risk category and then applies transformations from its open PyRIT framework to get past a model's surface alignment. You do not need their tooling to start, but you do need to treat your seed prompts as a living set that grows every time you find something new.

Two practical constraints shape where and how you run this. Run it in an isolated environment rather than production, because a real exercise generates genuinely harmful inputs and actions, and Microsoft recommends a purple environment, a non production setup built with production like resources, so that a successful attack causes no real damage. Who runs it matters as much as the tooling. The Center for Security and Emerging Technology notes that trust in a red teamer is built through education and career experience and community reputation, which is its way of saying that the creativity of the tester is the scarce input, not the tool they hold.

The severity rubric is the part that decides whether anyone acts on your report. The single most common reason a red team report gets ignored is that it mixes an embarrassing output with a real security breach and grades them the same, so the reader cannot tell the urgent from the awkward. Separate them explicitly. The table below is a rubric you can adopt as is, and the question in the last column is the one that matters, whether a boundary was actually crossed.

Grade every finding against that last column and the report sorts itself. A critical or high is work that blocks a deployment. A medium is a guardrail to tune. A low is a note for the content team. Conflating the four is how a genuinely dangerous finding ends up buried under a list of times the model was rude, which is the outcome the rubric exists to prevent. If your fix for the content policy failures is a filtering layer, be honest with yourself about what filtering can and cannot promise, a subject we treat in the piece on why guardrails reduce rates rather than guarantee.

How do you measure whether it is safe enough?

You need a number you can track across runs, and the field has settled on one. Microsoft's red teaming agent scores a system by its Attack Success Rate, the percentage of attacks that succeeded out of the total attempted, computed per risk category. A rate is more honest than a pass or fail, because red teaming rarely produces a clean pass, and it lets you watch the direction of travel as you fix issues and re-run. Pair the rate with the severity rubric, since an attack success rate that is low on criticals but higher on low severity content issues is a very different posture from the reverse. The number tells you how often you were breakable, and the rubric tells you how much each break mattered. Neither is meaningful alone, which is why a score card that reports both is what a deployment decision should rest on.

Where does automated red teaming stop being useful?

Sooner than the vendors imply, and knowing where is the difference between coverage and false confidence. Automation is genuinely valuable, because it broadens the range of attacks you can run and it scales, which is Microsoft's fourth lesson, that automation expands risk coverage. The catch is the scoring. The Center for Security and Emerging Technology found that over half of the tools it evaluated use AI powered scorers to judge outputs, and that those judges struggle with creativity, hallucinate, make biased judgments, and may not catch a mistake made by another AI. Automating the judge trades human control for speed, and you should document that cost rather than pretend it is free.

So the honest division of labour is this. Let automation do the breadth, running your seed set with every transformation across every category at a scale no human could match. Keep a human on the judgment, reviewing the findings the scorer flags and, more importantly, imagining the attacks the seed set never contained. Microsoft's fifth lesson states it plainly, that human expertise remains essential, and the reason is that a red team is a creative act, and creativity is exactly what the automated scorer lacks. The tool finds the variations of a known attack. The person finds the new one.

The last lesson is the one to end on, because it changes how you schedule the work. Microsoft's eighth lesson is that securing AI systems is never complete. A model updates, a new tool gets added, a document source changes, and yesterday's clean report no longer describes today's system. Red teaming is therefore not a gate you pass once but a scan you repeat, on a schedule and after every meaningful change, which is the same discipline that runs through our guide to the MaShop approach to security. Treat the first exercise as the start of a habit, not a certificate, and the method keeps paying off. If your next question is where AI helps test conventional systems rather than AI ones, that is a different job, and we grade it phase by phase in the piece on what AI penetration testing genuinely accelerates.