Red teaming for AI products: attacking your own system before others do

Red teaming means testing an AI product by trying to break it: prompt injection, data leaks and forbidden actions. The method and the first cases to test.

info Quick Definition

Red teaming for AI products means testing a system by attacking it on purpose: trying to make it ignore its instructions, reveal data it should not, or take actions it should not take. The goal is to find those failures before someone with bad intent does, or before a creative user does.

What is red teaming, and how is it different from a normal test?

NIST defines AI red-teaming as a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with its developers. A normal test checks that the system does what it should with expected inputs. Red teaming asks the opposite: what can the system do that it should not? It has an adversarial mindset: it assumes someone will push the limits and looks for ways to do it.

With language models, the attack surface is text. Any field the model reads, from a user message to a document in the knowledge base, is a possible entry point for an attack.

What types of red teaming exist?

The labs that practice it describe several approaches, and a product team can combine them at its own scale:

With domain experts. Anthropic describes testing with specialists on specific topics, such as child safety or election integrity, for risks a generalist team could not assess.
Open and community-based. Many people probe freely. In a 2022 paper, Anthropic released 38,961 attacks collected this way so other researchers can study them.
Automated. One model generates the attacks and you measure how many work. It scales better, but it only finds what its generator imagines.
External. OpenAI describes inviting people from outside the company, and explains that you have to decide who makes up the team, what access they get and what guidance they receive. An independent team reduces the bias of whoever built the feature.

For an AI product, a simple version is enough: a session with people from design, engineering and support, plus a list of cases you rerun on every model change.

Which risks should you test first?

OWASP publishes a list of the most common risks in LLM applications. For a product team, these are the ones tested most:

Risk (OWASP 2025)	What it means for a product	Typical test case
Prompt injection	A piece of text changes the model’s behavior	The user tries to make the assistant ignore its rules
Sensitive information disclosure	The model exposes data from other users or the system	Asking about orders from another account
Excessive agency	The model has more permissions than it needs	Asking it to delete a record it should not touch
Improper output handling	The response is used downstream without validation	Generated text that includes malicious code or links
System prompt leakage	The user gets the internal instructions	Asking the assistant to reveal its internal instructions
Unbounded consumption	A user generates huge costs	Very long inputs or loops

The first row is the one that shows up most in practice. OWASP distinguishes direct injection (the user writes the instruction) from indirect injection (the instruction is hidden in external content). OWASP’s example: a web page the model is asked to summarize carries hidden text that tells the model to insert a link leaking the conversation.

How do you run a red teaming exercise?

A practical four-step method:

Define what must not happen. For example: “does not reveal data from another account”, “does not send emails without confirmation”, “does not act outside the agent’s scope”.
Build a list of attacks. Use the categories in the table and add real support cases. Include variants: another language, encoded text, instructions hidden in a document.
Run and record. Test manually first, with someone who did not build the feature. Save every attempt and its result.
Turn findings into evals. Each attack that worked goes into your golden dataset as a regression case, so it does not come back unnoticed.

The exercise is continuous. An attack that failed today can work after a model or prompt change.

Which measures reduce the risk?

OWASP notes that it is unclear whether a foolproof method against prompt injection exists, but it proposes measures that reduce the impact:

Constrain the model’s role in the system prompt and tell it to ignore attempts to change its core instructions. It is one layer, not a guarantee.
Validate the output with code when it has a format: JSON, dates, identifiers. OWASP also suggests filtering inputs and outputs.
Enforce least privilege: tools use the application’s own permissions, not free access for the model.
Require human approval for high-impact actions.
Mark external content as untrusted so it carries less weight in the instructions.

OWASP’s own list of measures includes adversarial testing and attack simulation, which is what this exercise is. Claude Code follows a similar layered logic: its permission documentation says sandbox restrictions still apply even if a prompt injection bypasses Claude’s decision-making. The idea is that the barrier does not depend on the model deciding correctly.

When is this NOT the right depth?

When the product has no actions and no sensitive data. A chatbot that only answers public questions has a different and smaller risk.
When red teaming is done only with automated cases. Creative attackers find what a list does not cover.
When the result changes nothing in the product. A report of 40 attacks that nobody fixes is security theater.

Example

Illustrative example: a support application uses AI to look up orders and, with confirmation, issue refunds. The red teaming group, with one person from design and one from engineering, tries 60 cases. One of them: a customer pastes text that pretends to be an order from an administrator to issue a large refund. The model does not do it, because the refund requires a tool with code validation and human confirmation. But another case reveals a failure: asking it to reveal its instructions with a change of format makes the model reproduce part of the system prompt. The team adds the case to the golden dataset, rewrites the sensitive part of the prompt and removes internal rules that should not be there.

To see how this connects with agent controls, read guardrails and sandboxing and agent permissions. For the cases that feed the test, golden datasets. And for approving high-impact actions, approving an agent’s actions.

Common mistakes

Trusting that a system prompt saying “do not” works as a barrier.
Running red teaming only with English cases and no format variants.
Testing only user input and forgetting the external content the model reads.
Finding a failure and not turning it into a regression case.
Letting the same team that built the feature be the only one testing it.

Sources

Frequently asked questions

What is red teaming in AI?

Deliberately attacking an AI product to find security and behavior failures before a malicious user or a creative one does. It combines manual testing with automated cases.

What is prompt injection?

When a piece of text, either written by the user or coming from external content such as a web page or an email, changes the model’s behavior in unintended ways. OWASP lists it first (LLM01:2025) among the risks for LLM applications.

Can prompt injection be eliminated completely?

OWASP notes that it is unclear whether a foolproof method exists. What you can do is reduce the impact: limit permissions, require human approval for high-risk actions and treat external content as untrusted.

Who should do red teaming on a product?

Someone who did not build the feature, because they know its assumptions less well. Design, product and support can contribute real cases, but someone should try to break the system with an adversarial mindset.