How to red-team an AI agent on AWS: a repeatable method from scope to regression test

Security asks for a red-team report on the agent you are about to ship, and what exists is a list of jailbreak prompts someone pasted into the chat box. That list tests the model. You did not build the model and you cannot patch it. What you did build is the part an attacker needs: the tools, their arguments, the identities and the permissions. Those are what turn an injected sentence into a cancelled order or a leaked record.

If your agent runs on Amazon Bedrock or Amazon Bedrock AgentCore, reads outside text and has at least one tool that writes, the six-step method below fits it. The steps build on OWASP, NIST, MITRE ATLAS and AWS's own guidance. You also get what to test at each layer, how to report a result so the number means something, and a worked example on a public reference build.

I have done this kind of adversarial testing on production LLM applications and agents for clients. Those engagements are under NDA, so everything below stands on public standards, AWS documentation and one reference build anyone can run.

What the standards agree red teaming is

NIST's Generative AI Profile (AI 600-1) defines AI red-teaming as "a structured testing exercise used to probe an AI system to find flaws and vulnerabilities such as inaccurate, harmful, or discriminatory outputs." OWASP's GenAI Red Teaming Guide calls it "a structured methodology combining human expertise with automation and AI tools."

Three points from these documents shape the method:

• Results are statistical. OWASP says GenAI "involves probabilistic outputs, which requires statistically rigorous testing methods to assess vulnerabilities." One successful prompt proves a path exists. One failed prompt proves little.
• It is not a one-off. OWASP: "Red teaming is not a one-time event. Re-test after implementing fixes." The AWS Well-Architected Agentic AI Lens (AGENTSEC07-BP05) lists "Performing red team exercises only at initial deployment without scheduling regular assessments" as an anti-pattern.
• It is your job. The Amazon Bedrock prompt injection page says "the responsibility for secure application development and preventing vulnerabilities like prompt injection lies with the customer."

Microsoft's paper on red teaming 100 generative AI products draws a boundary in one of its lesson titles: "AI red teaming is not safety benchmarking." A benchmark score on the model tells you little about what your tools will do.

Notice that the deterministic layer is tested before anyone attacks the model, and that every finding comes back as a regression test before the next round.

Step 1: Scope the exercise from the agent's data flows

Start with an inventory, not a prompt list. For each tool the agent has, write down:

• who can put text into what it returns;
• what data it can read;
• whether it changes state or sends anything out.

Meta's "Agents Rule of Two" gives a simple way to read that inventory. Agents must satisfy "no more than two of the following three properties within a session". The three are processing untrustworthy inputs, reaching sensitive systems or private data, and changing state or communicating externally. Simon Willison's "lethal trifecta" makes the same point with private data, untrusted content and external communication. A session that has all three is where your red team spends most of its time.

OWASP's Top 10 for Agentic Applications 2026 adds a word for the design question this raises: "Least-Agency". Every tool you remove before the exercise is one less path to test.

Scoping also has a paperwork side. OWASP's guide says it should follow standards for "authorization for testing, data logging, reporting, deconfliction, communication/Opsec, and data dispositioning." Agree in writing which accounts, which data and which hours are in scope before anyone plants a payload.

Step 2: Map each threat to a named technique

Naming each threat after a published technique keeps the exercise honest and makes the report readable by a security team. Here is the mapping I would start from for an AWS agent. The ATLAS tactics are from release 2026.09:

OWASP draws the line between the second and fifth rows itself: ASI02 "covers cases where the agent operates within its authorized privileges but applies a legitimate tool in an unsafe or unintended way," and escalation belongs to ASI03.

For the AWS column, the AgentCore documentation gives a red team concrete things to try:

• The runtime's user ID header "does not verify the userId against an authenticated end-user identity," and the token vault uses that user ID as part of its binding key. Test whether one caller can reach another user's stored tokens.
• Putting AgentCore Gateway in front of the runtime "is only useful if callers can't reach the runtime directly bypassing the gateway." Test the direct path.
• In AgentCore Policy's LOG_ONLY mode "every tool call succeeds regardless of forbid policies," and "No separate action or condition key protects the mode field beyond bedrock-agentcore:UpdateGateway." Test who holds that permission.

Step 3: Test the deterministic layer first

Start with the controls that decide the same way every time: input validation, tool code, routing, IAM and policy. A failure there is a bug you can fix and pin with a test. It does not depend on how a model behaves today, and it is cheap to rerun.

The worked example is my public reference build, InfraTales/agentcore-mcp-gateway. It has two Lambda functions behind AgentCore Gateway as MCP tools, with Cognito client-credentials tokens. Cedar policies are meant to let only a token with the orders/admin scope call cancel_order; the first finding below shows where the match is looser than that. I added an adversarial suite to it: 44 tests that run with no AWS account.

The suite sends the tool calls an agent might write after reading an injected instruction, and checks what the deterministic layer does with them:

• Injected text in arguments: an instruction appended to an order ID, a newline, a wildcard, Arabic-Indic and fullwidth digits that look like an ID, an oversized ID, arrays and objects in place of an ID, an undeclared approved: true field and a __proto__ key. Every one returns INVALID_INPUT and no order changes.
• Routing: cancel_order sent to the read-only function, constructor as a tool name, a doubled ___ delimiter and a direct invoke without the gateway's context. Every one is refused before any tool runs.
• Policy decisions: the repository's actual Cedar statements, evaluated with the open-source Cedar authorizer against hostile requests. A read-only token, a missing scope, a tool no policy names and a different gateway ARN are all denied.

Two results are findings, and the suite records them as current behaviour:

1. The scope check is a substring match. The cancel policy uses like "*orders/admin*", so a token whose scope is legacy-orders/admin would be allowed to cancel. This stack's user pool cannot issue that scope. A second resource server in the same pool whose identifier ends in orders could.
2. The tools do not know the end customer. The gateway authorizes the calling agent, not the person it is acting for, so any agent allowed to read can read any customer's orders. An injected instruction that names another customer's order ID is not stopped by this layer.

This is the policy test for the first finding, from the repository:

it('a scope from another resource server ending in "orders" is ALLOWED', () => {
  expect(decide({ tool: CANCEL, scope: 'legacy-orders/admin' }).decision).toBe('allow');
});

Notice that the test asserts allow. It documents the risk so that a later change to the policy shows up as a failing test, and it gives the reviewer the exact request to reason about.

How do you know the new tests add anything? Mutation testing answers that. The repository's mutation script breaks one safeguard at a time and expects a test to fail. I added four mutants for the adversarial suite, and all four were caught. Two of them passed the other 61 unit, contract and infrastructure tests. One was an order ID pattern that is no longer anchored at the end, and the other a cancel reason that leaks back to read tools. Only the adversarial suite caught them.

This layer also has limits. No model is in the loop and nothing is deployed. AgentCore Policy runs its own engine, so the policy tests rely on the AWS docs' description of how token claims become principal tags.

Step 4: Attack through the model, with a stated budget

Now put the model back in. The question changes from "what does the code do with this call?" to "how often can someone get the model to make it?"

Three open-source tools cover most of this, and each reaches AWS differently:

None of the three documents a target for an AgentCore Runtime agent. For that you point each tool's generic HTTP or REST target at your agent's endpoint, which I have not tested.

Automation is a helper. Microsoft's lessons say these tools "should not be used with the intention of taking the human out of the loop." Anthropic writes that "Human security researchers consistently outperform automated systems at discovering creative attack vectors." Use the tools for volume and regression. Spend people on the attack paths from step 2.

Adaptive attackers change the numbers a lot, so the budget belongs in the plan:

• In NIST's agent hijacking evaluation, new attacks written for the target raised the attack success rate "from 11% for the strongest baseline attack to 81%," and repeating each attack 25 times raised the average "from 57% to 80%." That is one study on one benchmark, so read it as a direction, not a constant.
• Nasr and co-authors report adaptive attacks that "bypass 12 recent defenses" with success "above 90% for most," against defences that mostly reported near-zero rates.
• On Gray Swan's Agent Red Teaming benchmark, Anthropic reports one of its own models at "roughly 0.1% on single attempts, and around 5–6% after 100 adaptive attempts."

For the indirect path, plant harmless canary instructions in the content your tools return: a ticket body, a document chunk, a web page. The companion article on prompt injection through tool results explains why a Converse guardrail never sees that text.

Step 5: Report a number that means something

OWASP's main metric is the attack success rate (ASR), the "Percentage of adversarial inputs that successfully exploit vulnerabilities or elicit undesired behavior." OWASP also says it sets no thresholds yet: "In future we will add guidance on thresholds and interpreting the metrics."

A NeurIPS 2025 position paper on ASR explains why a bare percentage misleads. Its authors argue that conclusions drawn from ASR comparisons are often "not supported" by the evidence, and warn about "apples-to-oranges comparisons or low-validity measurements". Changing the aggregation rule alone can raise the number. So every ASR in a report should carry:

1. The threat model: who the attacker is and which channel they control.
2. The attempt budget: attempts per case, and whether attacks adapted between attempts.
3. The aggregation rule: success on any attempt, or the share of attempts that succeeded.
4. The sample size and an interval around the rate. Miller's paper on evaluation statistics covers standard errors and paired comparisons. Applying it to ASR is my choice, not something the paper prescribes.

A finding is more useful than a rate. For each one I would record:

• the technique ID;
• the layer it got through;
• the attempts and successes, and the budget;
• which control changed because of it;
• the regression test that now pins it.

AWS gives you three measuring tools for this step:

• AgentCore Policy LOG_ONLY. It records what a policy would have decided without blocking. If you have a labelled set of benign and malicious prompts, the docs suggest computing "precision and recall at each threshold value". They treat LogOnlyDecisionFlips as the signal for promoting a rule to enforcement.
• InvokeGuardrailChecks. Announced on 2026-06-16, this Bedrock Guardrails API is "detect-only": it returns a score per check and leaves the decision to you. AWS lists "after a tool returns a result" among the places to call it, and the prompt attack check scores JAILBREAK, PROMPT_INJECTION and PROMPT_LEAKAGE separately. It is not available in the Mumbai Region as of 2026-10-27.
• AgentCore Evaluations. The built-in evaluators cover quality, safety and tool use, and none of them targets prompt injection. Code-based evaluators run "your own AWS Lambda function" for deterministic checks. The Agentic AI Lens suggests running evaluations "before and after red team exercises" to see what the exercise found that your evaluators missed.

One caution: Bedrock's built-in "robustness" metric measures how output changes under "minor, semantic preserving perturbations," such as typos and casing. It is a quality check, not an adversarial test.

Step 6: Turn every finding into a regression test

If a red-team finding does not become a test, nothing catches it when it comes back. OWASP's activity list includes "Retesting post-remediation". AWS Prescriptive Guidance on generative AI operations puts "automated adversarial tests (red teaming)" in the CI/CD security stage. It says the process "must be a systematic and repeatable effort that aligns with a recognized framework."

In practice that means three habits:

• Every deterministic finding gets a unit or policy test, like the scope test above.
• Every model-level finding becomes a case in your eval set, run with a fixed attempt budget so the rate is comparable release to release. The article on agent evals as a CI release gate covers how to run those cases in CI.
• A mutation run checks that the tests catch the safeguard being removed. A test that cannot fail does not protect anything.

Then put the human-led exercise on a calendar. The Agentic AI Lens asks for "human-led red team exercises on a regular cadence," built from a scenario library where each scenario has a "description, expected detection mechanism, and success criteria."

When a different choice is better

• Every tool only reads, none sends anything out, and the client shows replies as plain text. Steps 3 and 4 shrink to the read paths and the rendering path (AML.T0077). A lighter exercise may be proportionate.
• You need an independent result. CISA describes AI red teaming as "the third-party safety and security evaluation of AI systems." If a regulator, customer or board needs independence, the team that built the agent should not be the only team that tests it.
• You only have time for one thing. I would spend it on step 3. A deterministic rule on tool arguments and a test that proves it holds will outlast any prompt list.

Sources

Pages were read on 2026-10-27 unless noted.

AWS documentation

• AGENTSEC07-BP05 Regular security assessments and red teaming, AWS Well-Architected Agentic AI Lens: regular human-led red teaming, the scenario library, evaluations before and after.
• Prompt injection security, Amazon Bedrock User Guide: customer responsibility.
• Include a guardrail with the Converse API, Amazon Bedrock User Guide: tool results are not evaluated.
• Use the InvokeGuardrailChecks API in your application, Amazon Bedrock User Guide: detect-only checks, where to call them, supported Regions.
• Concepts: Messages, content block types, and checks and Score definitions, Amazon Bedrock User Guide: prompt attack categories, scored per attack vector.
• Amazon Bedrock Guardrails announces a new API targeting agentic AI workflows, AWS What's New, 2026-06-16.
• Authenticate and authorize with Inbound Auth and Outbound Auth, Amazon Bedrock AgentCore Developer Guide: the unverified user ID header, the token vault binding key and the gateway bypass.
• Policy enforcement modes, Amazon Bedrock AgentCore Developer Guide: LOG_ONLY and UpdateGateway.
• Test a policy in LOG_ONLY mode, Amazon Bedrock AgentCore Developer Guide: LogOnlyDecisionFlips, precision and recall on a labelled set.
• Prompt templates and Evaluators, Amazon Bedrock AgentCore Developer Guide: built-in and code-based evaluators.
• Release notes for Amazon Bedrock AgentCore, Amazon Bedrock AgentCore Developer Guide: what the built-in evaluators cover.
• Review metrics for an automated model evaluation job in Amazon Bedrock (console), Amazon Bedrock User Guide: the robustness metric.
• Hardening the generative AI application through a GenAIOps framework, AWS Prescriptive Guidance: adversarial tests in CI/CD.

Standards and specifications

• GenAI Red Teaming Guide, OWASP Gen AI Security Project, 2025-01-23: definitions, probabilistic testing, scoping, retesting, ASR and the missing thresholds.
• OWASP Top 10 for Agentic Applications for 2026, OWASP Gen AI Security Project, 2025-12-09: ASI01, ASI02, ASI03, ASI06 and Least-Agency.
• LLM01:2025 Prompt Injection, OWASP Gen AI Security Project: indirect prompt injection.
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, NIST, 2024-07: the definition of AI red-teaming.
• MITRE ATLAS Data, MITRE: technique names and tactic mapping in release 2026.09.
• AI Red Teaming: Applying Software TEVV for AI Evaluations, CISA, 2024-11-26: red teaming as third-party evaluation.
• Technical Blog: Strengthening AI Agent Hijacking Evaluations, NIST, 2025-01-17, updated 2025-12-19: adaptive attacks and repeated attempts (single study).

Industry sources

• Lessons From Red Teaming 100 Generative AI Products, Microsoft AI Red Team, 2025-01-13: red teaming is not benchmarking; automation keeps humans in the loop.
• Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming, Chouldechova et al., NeurIPS 2025 position paper: why bare ASR comparisons mislead.
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, Evan Miller, 2024-11-01: standard errors and paired comparisons for evaluations.
• The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections, Nasr et al., 2025-10: adaptive attacks against 12 defences (abstract).
• Mitigating the risk of prompt injections in browser use, Anthropic, 2025-11-24: human researchers and automated attackers.
• How we contain Claude across products, Anthropic, 2026-05-25: single-attempt and adaptive attack rates on Gray Swan's benchmark (vendor self-report).
• Agents Rule of Two: A Practical Approach to AI Agent Security, Meta, 2025-10-31.
• The lethal trifecta for AI agents: private data, untrusted content, and external communication, Simon Willison, 2025-06-16 (personal blog of a practitioner).
• Red team Configuration, AWS Bedrock Agents and AWS Bedrock, promptfoo docs: plugins, strategies, the openai:gpt-5 default, framework IDs, Bedrock Agents and Knowledge Base providers.
• garak, LLM vulnerability scanner, NVIDIA: probes, detectors, the Bedrock and REST generators.
• Python Risk Identification Tool for generative AI (PyRIT), Microsoft, with its Framework doc and Release v0.10.0 notes: components, orchestrators renamed, the LiteLLM target.

Reference code

• InfraTales/agentcore-mcp-gateway: own reference build; the adversarial suite is in test/adversarial/ and documented in docs/adversarial-tests.md.