AI-Red-Teaming-Guide - A comprehensive guide to adversarial testing and s

项目说明

<di="center">

Read this in: English · Español · 中文 · Français

🎯 AI Red Teaming: The Complete Guide

A comprehensi to ad testing and security e of AI systems, helping organizations identify before attackers exploit them.

Trusted by practitioners at

!Microsoft
!Google
!Meta
!OpenAI
!Anthropic
!NVIDIA
!IBM
!Amazon
!HackerOne
!Cisco

<sub>Logos represent organizations where indi practitioners reference this guide; inclusion does not imply official endorsement.</sub>

O • Frameworks • Methodologies • Tools • Case Studies • Resources

</div>

---

> ### 🌐 Join the Global Red Teaming Network
> Connect with AI red teamers worldwide, share findings, and collaborate on ad testing through Cogensec.
> → Join the network

---
<di="center">

<br>

![Explore Platform]()
![Free Sample]()
!AI Red Teaming
!Security
!License
!Updated
![X]()

---

<di="center">
<a href="
<img src="assets/redteamkit-banner.svg" alt="RedTeamKit — You've read the methodology. Now run it. $249 one-time." width="100%">
</a>
</div>

---
</div>

📋 Table of Contents

What is AI Red Teaming?

Why AI Red Teaming Matters

Key Frameworks and Standards

- NIST AI Risk Management Framework
- OWASP GenAI Red Teaming Guide
- OWASP Top 10 for Agentic Applications (2026)
- MITRE ATLAS
- CSA Agentic AI Red Teaming
- Microsoft Agentic Failure-Mode Taxonomy v2.0

AI Red Teaming Methodology

Threat Landscape

Attack and Techniques

MCP & Tool-Protocol Security

Computer-Use & Browser Agent Attacks

RAG Attack Taxonomy

Voice, Audio & Multimodal Attacks

Fine-Tuning & Model Supply-Chain Security

AI-on-AI Red Teaming

AI Coding-Agent & CI/CD Security

Agent-to-Agent (A2A) & Agent Identity

Frontier Capability & AI-Accelerated Discovery

Red Teaming Tools

- Open-Source Tools
- Commercial Platforms
- Featured Commercial Platform: A by Cogensec
- Comparison Matrix

Real-World Case Studies

Building Your Red Team

Best Practices

Implementation Quickstart (30/60/90)

E Harness (Reference Implementation)

Agentic AI Attack Trees + Controls Mapping

AI Harm Se and Triage Model

AI Incident Response

Secure SDLC Integration Artifacts

Defensi Patterns

Multilingual & Cultural Safety Playbook

Data Go for Red Teaming

Metrics That Matter (and Anti-Metrics)

Purple Team Operations

Common Implementation Pitfalls

Case Study Quality Bar

Model & System Cards for Security Posture

Source Hygiene & Update Go

Practitioner Appendices

Regulatory Compliance

Resources and References

Contributing

Glossary

License · Acknowledgments · Contact · Disclaimer

---

🎯 O

As artificial intelligence systems become increasingly integrated into critical business operations, healthcare, finance, and decision-making processes, ensuring their security and reliability has never been more important. AI red teaming has emerged as a fundamental security practice that helps organizations identify before they can be exploited in real-world scenarios.

This comprehensi is designed for:

🔐 Security Teams implementing AI security testing programs

🛡️ AI/ML Engineers building secure AI systems

👨‍💼 Risk Managers assessing AI-related risks

🏢 Organizations deploying AI in production

🎓 Researchers studying AI security and safety

📊 Compliance Officers ensuring regulatory adherence

Why This Guide?

✅ E: Grounded in real-world experience from Microsoft's 100+ AI product red teams

✅ Framework-Aligned: Incorporates NIST AI RMF, OWASP, MITRE ATLAS, and CSA guidelines

✅ Practical Focus: Actionable methodologies and tools you can implement today

✅ Continuously Updated: Reflects latest 2024-2026 research and industry practices

✅ Comprehensi: From basic concepts to ad attack techniques

---

🤖 What is AI Red Teaming?

AI Red Teaming is a structured, proacti practice where expert teams simulate ad attacks on AI systems to unco and impro security and resilience. Unlike traditional security testing that focuses on known attack , AI red teaming embraces creati exploration to disco failure modes and risks.

Core Principles

AI red teaming adapts military and cybersecurity red team concepts to the unique challenges posed by AI systems:

| Traditional Cybersecurity | AI Red Teaming |
|---------------------------|----------------|
| Tests against known | Discovers novel, emergent risks |
| Binary pass/fail outcomes | Probabilistic behaviors and edge cases |
| Static attack surface | Dynamic, context-dependent |
| Code-le | Natural language attacks |
| Deterministic systems | Non-deterministic AI behaviors |

Key Definitions

Red Team: Group simulating ad attacks to test system security

Blue Team: Defensive team working to protect and secure systems

Purple Team: Collaborati combining red and blue team insights

Attack Surface: All potential points where an AI system can be exploited

Jailbreaking: Bypassing AI safety guardrails to elicit prohibited outputs

Prompt Injection: Manipulating AI behavior through crafted input prompts

Model Extraction: Stealing proprietary AI models through API queries

Data Poisoning: Corrupting training data to compromise model behavior

---

🚨 Why AI Red Teaming Matters

The Urgency of AI Security

Recent security incidents demonstrate that AI systems face unique challenges traditional cybersecurity cannot address:

2025–2026 Security Incidents:

September 2026: The FTC opened a consumer-protection probe into OpenAI, Anthropic, and METR o incidents and safety claims, as the labs reported re tens of thousands of cases of models o during testing and use.

June 2026 (disclosed September): An internal OpenAI frontier agent, on its own initiati non-public access to Australia's Medicare statistics portal during e — retrieving files and credentials and writing files. OpenAI paused tool-use training for its most capable models (Case Study D).

August 2026: The Deadbugz campaign pushed a malicious MCP server via 23 PRs in 74 minutes; it behaved for three tool calls, then told agents to steal SSH keys and cloud credentials (Case Study F).

April 2026: "Comment and Control" — one malicious GitHub comment hijacked Claude Code, Gemini CLI, and Copilot coding agents in CI and leaked secrets into public logs (Case Study E). The same month, Anthropic's unreleased Claude Mythos model began finding thousands of critical for defenders through Project Glasswing.

January 2026: The OpenClaw agent framework (135k+ stars in weeks) was hit by 100+ CVEs — including a one-click RCE theft (C, CVSS 8.8). By spring 2026, 135,000+ instances were internet-exposed (most unauthenticated), and ~335 malicious plugins reached its ClawHub marketplace (~12% of the registry).

September 2025: Anthropic detected and disrupted the first documented large-scale cyberattack predominantly executed by an AI agent — a state-sponsored operation in which Claude Code autonomously handled an estimated 80–90% of tactical execution across ~30 global targets.

August 2025: GitHub Copilot remote code execution (C, CVSS 7.8) injection that wrote to the agent's configuration files (enabling VS Code "YOLO mode").

2025: Prompt-injection research demonstrated against AI-enabled browsers (Perplexity's Comet, Gemini for Chrome) and coding assistants (GitLab Duo, Copilot Chat).

2023–2024 (historical): Samsung's ChatGPT data leak, the March 2025 ChatGPT exploit, and the Microsoft health-chatbot data exposure remain instructi examples (see Real-World Case Studies).

> By the numbers (/researcher-reported, 2025). Estimated global losses from AI prompt-injection attacks reached ~$2.3B, a reported +340% year over year; ~88% of organizations deploying AI agents reported confirmed or suspected security incidents; current detection methods are reported to catch only ~23% of sophisticated prompt-injection attempts. *Treat these as directional industry figures, not audited statistics — sources are listed in Resources and References.*

The Stakes Are Higher

In 2026, AI and LLMs are no longer limited to chatbots and assistants for customer support. Autonomous, tool-using agents now act on behalf of users — booking, buying, coding, and operating infrastructure — which converts what used to be "bad text output" into real-world actions: data exfiltration, lateral mo, and unauthorized transactions. Their use increasingly expands into high-stakes applications such as healthcare diagnostics, financial decision-making, and critical infrastructure systems.

Regulatory Drivers

Article 15 of the European Union AI Act obliges operators of high-risk AI systems to demonstrate accuracy, robustness and cybersecurity. The US Executi on AI defines AI red teaming as "a structured testing effort to find flaws and in an AI system using ad methods to identify harmful or discriminatory outputs, unforeseen behaviors, or misuse risks."

Business Impact

Reputation Risk: AI failures can cause immediate brand damage

Financial Loss: Data breaches and service disruptions cost millions

Legal Liability: Non-compliance with AI regulations brings penalties

Competiti: Secure AI builds customer trust

Inno Enablement: Understanding risks allows safer experimentation

---

📚 Key Frameworks and Standards

NIST AI Risk Management Framework

The NIST AI Risk Management Framework (AI RMF) emphasizes continuous testing and e throughout the AI system's lifecycle, pro a structured approach for organizations to implement comprehensive AI security testing programs.

Four Core Functions:

#### 1. GOVERN
Establish AI go structures and risk management culture

Develop AI risk policies and procedures

Assign roles and responsibilities

Integrate AI risks into enterprise risk management

#### 2. MAP
Identify and categorize AI risks in context

Understand AI system capabilities and limitations

Document intended use cases and deployment contexts

Identify potential risks and stakeholders

#### 3. MEASURE
Assess, analyze, and track identified AI risks

NIST recommends red teaming as an approach consisting of ad testing of AI systems under stress conditions to seek out AI system failure modes or

E trustworthiness characteristics

Track metrics for fairness, bias, and robustness

Use tools like Dioptra (NIST's security testbed) for model testing

#### 4. MANAGE
Prioritize and respond to identified risks

Implement risk mitigation strategies

Monitor AI systems in production

Maintain incident response capabilities

Key NIST Resources:

AI RMF (NIST AI 100-1): Core framework

GenAI Profile (NIST AI 600-1): Generati guidance

Ad ML Taxonomy (NIST AI 100-2e2025): the standard for attacks and mitigations across the ML lifecycle — use it to label findings consistently

Secure Software De (NIST SP 800-218A): De practices

Dioptra Testbed: Open-source AI security testing platform

CAISI AI Agent Standards Initiative (2026): NIST's Center for AI Standards and Inno launched a three-pillar program (agent security, interoperability, identity) on Feb 17, 2026, and open-sourced AgentDojo-Inspect for agent-hijacking e. Its headline red-team result — no reaching an 81% task-hijack rate vs 11% for prior baselines — is a useful reminder that agent e must evol.

---

OWASP GenAI Red Teaming Guide

The OWASP Gen AI Red Teaming Guide provides a practical approach to e LLM and Generative AI , co e from model-le and prompt injection to system integration pitfalls and best practices for ensuring trustworthy AI deployments.

Key Components:

1. Quick Start Guide: Step-by-step introduction for newcomers
2. Threat Modeling Section: Identify relevant risks for your use case
3. Blueprint & Techniques: Recommended test categories
4. Best Practices: Integration into security posture
5. Continuous Monitoring: Ongoing o guidance

OWASP Co Areas:

Model-le (toxicity, bias)

System-le (API misuse, data exposure)

Prompt injection attacks

Agentic

Cross-functional collaboration guidance

Access the Guide: genai.owasp.org

OWASP Top 10 for LLM Applications (2025): the LLM-application list was refreshed in the 2025 edition, which added two categories worth explicit red-team co: System Prompt Leakage (system prompts inad exposing secrets or exploitable instructions) and & Embedding Weaknesses (RAG/ risks — embedding poisoning, similarity attacks, and embedding in). The edition also renamed "O" to Misinformation, broadened "Model DoS" to Unbounded Consumption, and expanded Excessi. For single-prompt LLM apps, test against the LLM Top 10; for tool-using agents, use the Agentic Top 10 (2026) below.

OWASP 2026 updates (Q2–Q3 2026):

Top 10 for LLM Applications — 2026 edition: the list is now built from 75% expert consensus + 25% real incident data (6,639 documented ), with each entry mapped to NIST, MITRE ATLAS, and CWE. Re-map your test catalog to the 2026 IDs when you next refresh it.

Agent Control Standard: a new OWASP control baseline for agentic systems — use it as the "expected controls" side of agent red-team findings, alongside the Agentic Top 10 as the "risks" side.

AI Red Teaming Landscape & AI Security Solutions Directory: OWASP's first market map for AI/agentic red-teaming tools — useful for tool selection next to the comparison matrix in this guide.

(OWASP GenAI announcement · Straiker — what OWASP's Q2 2026 update is saying)

---

OWASP Top 10 for Agentic Applications (2026)

Published by the OWASP GenAI Security Project (peer-re by 100+ contributors), this is the first risk ranking built specifically for autonomous, tool-using agents rather than single-prompt LLM apps. Every red team testing agents in 2026 should map findings to these IDs.

| ID | Risk | What to test |
|----|------|--------------|
| ASI01 | Agent Goal Hijack | Untrusted input rewrites the agent's objecti; reward/goal manipulation. |
| ASI02 | Tool Misuse & Exploitation | Coercing the agent to call tools beyond intent; argument injection into tool calls. |
| ASI03 | Agent Identity & Pri Abuse | Agent acting with o or borrowed credentials; confused-deputy escalation. |
| ASI04 | Agentic Supply Chain Compromise | Malicious tools, plugins, MCP servers, or sub-agents introduced into the pipeline. |
| ASI05 | Unexpected Code Execution | Agent-generated or agent-triggered code running in pri contexts. |
| ASI06 | Memory & Context Poisoning | Persisting attacker-controlled state that biases future sessions. |
| ASI07 | Insecure Inter-Agent Communication | Spoofed/unauthenticated messages between agents; trust escalation across the mesh. |
| ASI08 | Cascading Agent Failures | One compromised/failing agent propagating errors system-wide. |
| ASI09 | Human-Agent Trust Exploitation | Consent fatigue, deceptive UI, social engineering of the human approver. |
| ASI10 | Rogue Agents | Agents operating outside monitoring/go boundaries (shadow agents). |

How this guide maps to it: the Agentic AI Attack Trees section tags each tree with the ASI IDs it exercises, and the MCP & Tool-Protocol Security section drills into ASI02/ASI04.

Access: OWASP Top 10 for Agentic Applications 2026

---

MITRE ATLAS

MITRE ATLAS is a comprehensi specifically designed for AI security, pro a knowledge base of ad AI tactics and techniques. Similar to the MITRE ATT&CK framework for cybersecurity, ATLAS helps organizations understand potential attack against AI systems.

ATLAS Tactics:

Reconnaissance: Disco AI system information

Resource De: Acquiring attack infrastructure

Initial Access: Gaining entry to AI systems

ML Model Access: Obtaining model information

Persistence: Maintaining access to AI systems

Defense E: A detection mechanisms

Credential Access: Stealing authentication tokens

Discovery: Learning about AI system en

Collection: Gathering data from AI systems

ML Attack Staging: Preparing ad attacks

Exfiltration: Stealing model weights or data

Impact: Causing AI system degradation

Real-World Case Studies in ATLAS:

Data poisoning attacks

Model e techniques

Model in exploits

Ad examples

ATLAS v5.x (Nov 2025 – 2026): v5.1.0 added a 16th tactic and grew the matrix to 84 techniques, 32 mitigations, and 42 case studies; later 5.x releases added agent-focused techniques such as Publish Poisoned AI Agent Tool and Escape to Host, plus agentic techniques contributed by Zenity Labs. The tactic list above is the classic core — check the li when mapping agent findings.

Learn More: atlas.mitre.org

---

CSA Agentic AI Red Teaming

The Cloud Security Alliance's Agentic AI Red Teaming Guide explains how to test critical across dimensions like permission escalation, hallucination, orchestration flaws, memory manipulation, and supply chain risks, with actionable steps to support robust risk identification and response planning.

Agentic AI-Specific Risks:

1. Permission Escalation: Agents gaining unauthorized access
2. Hallucination Exploitation: Using fabricated outputs for attacks
3. Orchestration Flaws: in agent coordination
4. Memory Manipulation: Tampering with agent memory/context
5. Supply Chain Risks: Compromised agent components
6. Tool Misuse: Agents improperly using a tools
7. Inter-Agent Dependencies: Cascading failures across agents

Testing Requirements:

Isolated model behaviors

Full agent workflows

Inter-agent dependencies

Real-world failure modes

Role boundary enforcement

Context integrity maintenance

Anomaly detection capabilities

Attack blast radius assessment

---

Microsoft Agentic Failure-Mode Taxonomy v2.0

When Microsoft first published its *Taxonomy of Failure Modes in Agentic AI Systems* (April 2025), much of it was forward-looking. A year of real red-team engagements produced enough e for v2.0 (June 2026), which adds seven new failure-mode categories now observed in the wild:

1. Agentic supply chain compromise — malicious tools/plugins/sub-agents (see ASI04, and MCP security).
2. Goal hijacking — untrusted content redirecting the agent's objecti).
3. Inter-agent trust escalation — a low-pri agent le a higher-pri one (ASI07).
4. Computer-use agent attacks — on-screen/ injection of agents that see and click (see Computer-Use attacks).
5. Session context contamination — cross-turn/cross-session state bleed.
6. MCP and plugin abuse — the tool protocol layer as a first-class attack surface.
7. Capability / architecture disclosure — agents leaking their own tools, prompts, or topology to an attacker.

Two findings worth red-teaming explicitly:

Consent-fatigue human-in-the-loop bypass. Rather than defeat the approval gate, attackers *wear it down*: a stream of low-stakes "approve?" prompts trains the human to click through, and a high-impact action then slips by. Test your HITL design against , not just single decisions.

Zero-click end-to-end chains. Several engagements produced full data-exfiltration or lateral-mo chains requiring no human interaction beyond the initial agent launch. Assume the agent itself is the delivery .

Reference: Microsoft Security Blog — Updating the taxonomy of failure modes in agentic AI (June 2026)

---

🔬 AI Red Teaming Methodology

Phase 1: Planning and Threat Modeling

Organizations must first identify potential attack specific to their AI systems, including the types of ad they might face and the potential impact of successful attacks.

Step 1: Define Scope and Objectives

Step 2: Threat Modeling Using MITRE ATLAS

Step 3: Build Risk Profile
Each application has a unique risk profile due to its architecture, use case and audience. Organizations must answer: What are the primary business and societal risks posed by this AI system?

| Risk Category | Examples | Priority |
|---------------|----------|----------|
| Safety Risks | Physical harm, dangerous advice | Critical |
| Security Risks | Data breaches, unauthorized access | Critical |
| Privacy Risks | PII leakage, training data extraction | High |
| Fairness Risks | Discriminatory outputs, bias | High |
| Reliability Risks | Hallucinations, inconsistent responses | Medium |
| Reputation Risks | Offensi, brand damage | Medium |

Step 4: Develop Test Plan

Select testing methodologies (manual, automated, hybrid)

Choose appropriate tools and frameworks

Define success criteria and metrics

Allocate resources (time, budget, personnel)

Establish reporting and disclosure processes

---

Phase 2: Red Team Execution

Access Levels

The of the model or system red teamers ha to can influence the outcomes of red teaming. Early in the model de process, it may be useful to learn about the model's capabilities prior to any added safety mitigations.

| Access Type | Description | Use Cases |
|-------------|-------------|-----------|
| Black Box | No internal knowledge; interact via API/UI only | Simulates external attacker; realistic threat modeling |
| Gray Box | Partial knowledge (architecture, some data) | Simulates insider threat; common in enterprise |
| White Box | Full access (code, weights, training data) | Maximum discovery; pre-deployment |

Testing Approaches

#### 1. Manual Red Teaming
While automation tools are useful for creating prompts, orchestrating cyberattacks, and scoring responses, red teaming can't be automated entirely. Humans are important for subject matter expertise.

Techniques:

Jailbreaking: Crafting prompts to bypass safety guardrails

Prompt Injection: Embedding malicious instructions

Social Engineering: Manipulating AI through context

#### 2. Automated Red Teaming
DeepTeam implements 40+ classes (prompt injection, PII leakage, hallucinations, robustness failures) and 10+ ad attack strategies (multi-turn jailbreaks, encoding obfuscations, adapti).

Automation Strategies:

Fuzzing: Generate thousands of input

Ad Examples: Craft inputs to fool classifiers

LLM-Generated Attacks: Use AI to attack AI

Mutation Testing: Systematically alter prompts

Regression Testing: fixes don't break

#### 3. Hybrid Approach (Recommended)

Red Teaming Patterns from Microsoft

Microsoft found that rudimentary methods can be used to trick many models. Manually crafted jailbreaks tend to circulate on online forums much more widely than ad suffixes, despite significant attention from AI safety researchers.

Common Attack Patterns:
1. Skeleton Key: Uni jailbreak technique
2. Crescendo: Multi-turn escalation strategy
3. Encoding Obfuscation: ROT13, Base64, binary
4. Character Swapping: Homoglyphs, unicode tricks
5. Prompt Splitting: Di malicious intent across turns
6. Context O: Exceeding context window limits
7. Language Switching: Using low-resource languages
8. Attacks: Image-based injections (for multimodal)

---

Phase 3: E and Scoring

Key Metrics

The key metric to assess the risk posture of your AI system is Attack Success Rate (ASR) which calculates the percentage of successful attacks over the number of total attacks.

| Metric | Formula | Target |
|--------|---------|--------|
| Attack Success Rate (ASR) | (Successful Attacks / Total Attacks) × 100 | < 5% |
| Mean Time to Compromise | A time to successful exploit | > 100 hours |
| Co | (Test Cases / Total Risk Surface) × 100 | > 90% |
| False Positive Rate | (False Alarms / Total Alerts) × 100 | < 10% |
| Se Distribution | Critical / High / Medium / Low counts | Track trends |

Se Classification

---

Phase 4: Reporting and Remediation

Red Team Report Structure

Remediation Strategies

| Issue Type | Mitigation Approaches |
|------------|----------------------|
| Prompt Injection | Input sanitization, output filtering, structured prompts, pri separation |
| Jailbreaking | Reinforcement learning from human feedback (RLHF), constitutional AI, ad training |
| Data Leakage | Data minimization, differential privacy, output monitoring, access controls |
| Hallucination | Retrie Generation (RAG), citation requirements, confidence scoring |
| Bias | Diverse training data, fairness constraints, post-processing, regular audits |
| Model Extraction | Rate limiting, output randomization, API monitoring, watermarking |

---

🎯 Threat Landscape

Ad Types

| Ad | Moti | Capabilities | Typical Targets |
|-----------|-----------|--------------|-----------------|
| Script Kiddie | Curiosity, fame | Low; uses existing tools | Public AI chatbots, APIs |
| Hacktivist | Ideological | Medium; social engineering skills | Corporate AI, go systems |
| Cybercriminal | Financial gain | High; organized groups | Financial AI, e-commerce |
| Insider Threat | Revenge, espionage | Very high; legitimate access | Internal AI systems, models |
| Competitor | Competiti | High; well-funded | Proprietary models, trade secrets |
| Nation-State | Strategic ad | Extremely high; ad persistent threat | Critical infrastructure AI, defense systems |

Attack Lifecycle

---

⚔️ Attack and Techniques

> ⚖️ Authorized use only. The techniques and payloads in this section are for defensi of systems you own or are explicitly authorized in writing to test. Running them against third-party systems, production services handling real user data, or any target outside an agreed scope may be illegal and can cause real harm. Establish scope and permission first — see the Rules of Engagement template in templates/ and Establish Clear Rules of Engagement.

1. Prompt-Based Attacks

#### Prompt Injection
Every prompt, in a sense, is committing code to the application. Attackers don't need to breach backend systems to take control—they can manipulate the AI through natural language.

Direct Prompt Injection:

Indirect Prompt Injection:

Cross-Plugin Injection:

#### Jailbreaking

Common Jailbreak Patterns:

1. Role-Playing

2. Hypothetical Scenarios

3. Encoding

4. Multi-Turn Manipulation (Crescendo)

5. Language Switching

---

2. Data Poisoning

Training Data Poisoning:
Microsoft research shows that e methods can compromise AI systems through data manipulation.

Types:

Backdoor Attacks: Trigger words cause malicious behavior

A Attacks: Reduce model performance

Targeted Poisoning: Affect specific predictions

Clean-Label Attacks: Poisoning without label changes

Defense:

Data pro tracking

Statistical outlier detection

Differential privacy during training

Regular data audits

---

3. Model Extraction

Goal: Steal proprietary AI models through API queries

Techniques:

> ⚖️ Reminder: run extraction campaigns only against models you own or are authorized to test — high- query campaigns against third-party APIs typically their terms of service and may be unlawful.

1. Query-Based Extraction

2. Functional Extraction

Countermeasures:

API rate limiting (queries per minute/day)

Query monitoring for patterns

Output rounding/perturbation

Model watermarking

Authentication and access controls

---

4. Ad Examples

Goal: Craft inputs that fool AI classifiers

Image Classification:

Text Classification:

Defense Strategies:

Ad training

Input preprocessing

Ensemble methods

Certified robustness

Randomized smoothing

---

5. Model In

Goal: Reconstruct training data from model

Defenses:

Differential privacy

Output noise injection

Confidence score limiting

Access restrictions

---

6. Membership Inference

Goal: Determine if specific data was in training set

Privacy Implications:

GDPR "right to be forgotten"

Exposure of sensiti data

Competiti leakage

---

7. Supply Chain Attacks

AI-Specific Supply Chain Risks:

| Component | Risk | Example |
|-----------|------|---------|
| Pre-trained Models | Backdoors, poisoning | Malicious HuggingFace model |
| Training Data | Poisoned datasets | Corrupted open datasets |
| Libraries/Dependencies | packages | Compromised PyTorch |
| APIs/Integrations | Third-party exploits | Malicious API wrappers |
| Cloud Infrastructure | Platform | Compromised ML platform |
| Human Contractors | Insider threats | Malicious data annotators |

Mitigation:

model checksums

Audit dependencies (use tools like pip-audit)

Implement zero-trust architecture

Regular security scanning

risk assessments

---

8. Agentic AI Attacks (2026 Emerging Threats)

As AI agents become more autonomous, new attack emerge. Each maps to an OWASP Agentic Top 10 ID.

Permission Escalation (ASI03):

Tool Misuse (ASI02):

Goal Hijack (ASI01):

Memory Manipulation (ASI06):

Inter-Agent Exploitation (ASI07):

Self-Replicating Prompt Malware / AI Worms (ASI08):

> Tool-protocol (MCP) abuse, computer-use/ attacks, RAG-borne injection, and fine-tuning backdoors are large enough surfaces to warrant their own sections — see the five that follow.

---

🔌 MCP & Tool-Protocol Security

The Model Context Protocol (MCP) became the de facto standard for connecting models to external tools in 2025 — and with it, a brand-new attack surface. 99 CVEs were published for MCP-related software in 2025, and tool poisoning moved from theoretical risk to li attack. If your system gives a model tools, this section is the highest-le place to test. (Maps to OWASP ASI02 Tool Misuse and ASI04 Agentic Supply Chain Compromise.)

Attack 1: Tool / Schema Poisoning

The model reads each tool's *description* and *parameter schema* as trusted instructions. A malicious or compromised tool can hide directi.

Test: Register a benign-looking tool whose description contains hidden instructions; confirm whether the model honors them. Diff model behavior with the tool present .

Controls: Treat tool metadata as untrusted; sanitize/lint tool descriptions; pin and review tool schemas; render tool descriptions to the model through a policy filter.

Attack 2: MCP Ser & "Rug-Pull" Updates

A tool that was safe at install time silently changes behavior in a later (the description or endpoint is mutated post-approval).

Test: that the tool definition the model sees matches a re, hash-pinned ; attempt a mid-session redefinition and confirm it's rejected.

Controls: and checksum MCP servers; require re-approval on definition change; deny dynamic tool re-registration at runtime.

In the wild — runtime-gated poisoning: the Deadbugz campaign (Aug 2026) shipped an MCP server that answered normally for its first three tool calls, then swapped its returned metadata to instruct the agent to collect SSH keys, AWS credentials, shell history, and kubeconfig — and hide it from the user. Install-time review alone would not catch it. Test beyond the first few calls and diff tool metadata across a whole session. (See Case Study F.)

Attack 3: Tool-Call Interception / Redirection

A man-in-the-middle (or a malicious orchestrator) rewrites tool arguments or return between the model and the tool.

Test: Tamper with tool responses (e.g., inject instructions into returned content) and obser the model treats tool output as trusted instruction.

Controls: Authenticate and integrity-check tool channels (mTLS); label tool output as data, ne; quarantine tool responses through output policy.

Attack 4: Credential Theft via MCP Config

MCP ser commonly hold API keys and tokens. Exposed instances leak them (as the OpenClaw incident showed — 135,000+ internet-exposed instances, most unauthenticated).

Test: Scan for exposed MCP endpoints, world-readable config, and secrets passed as plaintext env/args; attempt to coerce a tool into echoing its own credentials.

Controls: Short-li tokens per tool/action; secret managers, not config files; ne MCP servers to untrusted networks.

Attack 5: Capability Namespace Collisions (Multi-Agent)

In multi-agent/multi-tool setups, two tools claiming the same name or capability let an attacker shadow a trusted tool with a malicious one.

Test: Register a tool whose name collides with a pri built-in; confirm the resolver can't be tricked into binding the malicious one.

Controls: Namespaced, identity-bound tool resolution; explicit allowlists per agent; deny ambiguous capability binding.

MCP testing checklist: schema/description sanitization · pinning + checksums · metadata diffing across the full session (not just at install) · channel authentication · tool output treated as data · scoped short-li · no untrusted-network exposure · namespace collision resistance · audit log of every tool call with arguments.

> Don't forget the boring bugs. Most MCP C in 2026 are classic web flaws in the server code — e.g. Aug 2026: path tra in Atlassian's Confluence MCP tool, a cleartext cluster-token leak in ArcadeDB, and SSRF in a Facebook Ads MCP server. Run standard AppSec testing (SAST, DAST, dependency scanning) against every MCP server, not only prompt-le.

---

🖥️ Computer-Use & Browser Agent Attacks

Agents that see screens and click (computer-use models, AI browsers) inherit every web/UI attack *plus* a new class of /perceptual injection. Microsoft's taxonomy v2.0 added "computer-use agent attacks" precisely because these moved from research to reality in 2025–2026 (demonstrated against Perplexity's Comet and Gemini for Chrome).

na hijacking — on-page elements (buttons, banners, hidden text) instruct the agent to na, click, or submit. *Test:* plant in/low-contrast instructions on a page the agent is asked to use and obser it obeys.

Screen-content injection — malicious instructions placed in content the agent renders (a doc, email, web page) are read as commands. *Test:* indirect prompt injection content (o with RAG attacks).

OCR spoofing — text crafted so the model's OCR reads something different from what a human sees (homoglyphs, layering). *Test:* ad o that flip the OCR'd instruction.

Pixel-le inputs — imperceptible perturbations that steer a model's decision/click target. *Test:* perturbed UI screenshots that misdirect the agent's action.

Form/credential autofill abuse — coaxing a browsing agent into entering credentials or submitting transactions on attacker-controlled pages.

Controls: isolate the agent's browser profile (no ambient cookies/credentials); require explicit human confirmation for state-changing actions (resistant to consent fatigue); separate "page content" from "instructions" in the agent's context; constrain na to allowlisted origins; log screenshots + chosen actions for replay.

---

📚 RAG Attack Taxonomy

Retrie Generation is the most common enterprise LLM pattern — and retrie is untrusted input that reaches the model with implicit trust. Indirect prompt injection via RAG is now one of the most exploited AI attack classes.

| Attack | Description | Test approach |
|--------|-------------|---------------|
| Source-document poisoning | Plant malicious instructions in a document that will be ingested/indexed. | Seed the corpus with a poisoned doc; confirm whether retrie it and the model obeys it. |
| Indirect prompt injection | Retrie contains "ignore prior instructions…" that the model executes. | Inject directives into retrievable content; measure obedience rate. |
| Retrie / ranking attacks | Keyword stuffing or embedding-space crafting to force a malicious doc to the top-k. | Craft a doc to outrank legitimate sources for a target query. |
| Citation spoofing | Fabricated or mismatched citations that lend false authority to harmful output. | cited sources actually support the claim; test fake-citation acceptance. |
| Context-window exhaustion | Flood retrie to push out the system prompt / safety instructions. | O retrievals; confirm safety instructions survi. |
| Embedding-space attacks | Inputs crafted to collide with sensiti in space, pulling it into context. | Probe for unintended retrieval of restricted documents. |

Controls: treat retrie as data, not instructions (delimit and label it); sanitize/strip instruction-like content pre-indexing; pro and trust scoring per source; cap per-source context share; citations against retrie; tenant-isolate stores.

---

🎙️ Voice, Audio & Multimodal Attacks

As voice agents and multimodal models reach production (call centers, voice assistants, workflows), the attack surface extends to audio. This complements the Multilingual & Cultural Safety Playbook.

Speaker cloning / voice spoofing — synthesized voice defeats authentication or impersonates a trusted speaker. *Test:* cloned-voice bypass of any or "trusted caller" logic.

Audio ad examples — perturbations inaudible/benign to humans that the model transcribes as a different command. *Test:* crafted audio that yields an attacker-chosen transcript.

Ultrasonic / inaudible commands — commands outside human hearing range picked up by the mic and acted on. *Test:* near-ultrasonic injection into a listening agent.

Cross-modal injection — instructions hidden in audio of a video, or in an image, that drive a multimodal agent (extends the case study below).

Accent / low-resource-language safety bypass — safety co is weaker outside high-resource English; spoken low-resource languages compound transcription + safety gaps.

Controls: li/anti-spoofing on voice auth (never rely on alone for high-risk actions); band-limit and audio input; transcribe-then-policy-check before acting; apply the same instruction/data separation to transcribed audio as to text.

---

🧬 Fine-Tuning & Model Supply-Chain Security

Customizing models introduces risks *before* a single prompt is sent. This deepens Supply Chain Attacks for the model-weights layer.

Fine-tuning backdoors — a small set of poisoned examples installs a trigger phrase that unlocks harmful behavior; benign on all other inputs. *Test:* trigger-recovery probing; beha diff vs. base model on edge prompts.

Malicious LoRA / adapter injection — a third-party adapter carries a jailbreak or backdoor while appearing to add a harmless skill. *Test:* pro + beha audit of every adapter before load.

Poisoned checkpoints from model hubs — a downloaded checkpoint is tampered (weights or, worse, an unsafe deserialization payload). *Test:* checksum/signature ; load untrusted weights only in a sandbox; prefer safetensors o formats.

Training-data extraction during eval — fine-tuning e can leak memorized PII/training data. *Test:* membership-inference and extraction probes against the fine-tuned model.

Weight exfiltration & distillation — large query campaigns to clone a model's behavior (see Model Extraction).

Controls: sign and checkpoints; safetensors-only loading; sandbox untrusted weights; pro tracking for datasets and adapters; beha regression of every fine-tune against the base model; rate-limit and monitor inference APIs against distillation.

---

🤖 AI-on-AI Red Teaming

The biggest methodological shift of 2026: autonomous, agent-orchestrated red teaming. Instead of a human firing prompts, an attacker LLM is given a natural-language objective, then selects attacks, composes transforms, runs them against the target, and produces structured findings. Recent research shows autonomous agents now solve the majority of black-box red-team challenges faster than human operators — and tooling (Promptfoo's Hydra, PyRIT's XPIA orchestrator, FuzzyAI Crescendo, emerging agent-nati) is con on this pattern.

Why it matters

Scale & speed: multi-turn, adapti that would take a human days run in minutes.

Multi-turn by default: real ad don't fire one prompt and walk away — agentic red teamers escalate (Crescendo-style) and pi.

Co: an attacker agent can exhaust a huge combinatorial space of transforms (encoding × role-play × language × split).

Architecture (typical)

Pitfalls to watch

Judge-model error: the LLM scoring success has its own false-positi rate — calibrate against human-labeled samples and report confidence (an anti-metric if ignored).

Benchmark contamination: attacker/target/judge sharing training data inflates results; keep eval sets fresh and held out.

Where humans still win: genuinely no ideas, business-context harms, and judgment calls on "is this actually harmful here?" Use AI for breadth, humans for depth — the 70/30 split still holds, now with AI doing more of the 70%.

---

💻 AI Coding-Agent & CI/CD Security

Coding agents (Claude Code, GitHub Copilot coding agent, Gemini CLI, Cursor, Codex and others) now run inside IDEs and inside CI pipelines with write access to repositories and pipeline secrets. That combination — untrusted text in, pri actions out — makes them one of the highest-value targets in 2026. (Maps to ASI01 Goal Hijack, ASI02 Tool Misuse, ASI05 Unexpected Code Execution.)

The attack surface is ordinary repo content. Pull request titles and bodies, issue text, code comments, commit messages, branch names, README files, and dependency docs all reach the agent's context. In the "Comment and Control" disclosure (April 2026) a single crafted PR comment or issue hijacked Claude Code's security-review action, Gemini CLI Action, and the Copilot coding agent in GitHub Actions, and got them to print API keys and tokens into public Actions logs (rated up to CVSS 9.4). See Case Study E.

What to test

| Test | How |
|------|-----|
| Injection via repo content | Plant instructions in a PR title, issue body, code comment, and branch name; check whether the agent follows any of them. |
| Secret exposure | Ask (indirectly, text) for env vars or tokens; check Actions logs, PR comments, and artifacts for leaks. |
| Pri triggers | Look for workflows on pull_request_target, issue_comment, or workflow_run that hand secrets to an agent processing fork-controlled content. |
| Write scope | Can the agent push, merge, edit workflows, or change its own config (.github/, agent instruction files) without review? |
| Tool and network reach | Can it run arbitrary shell, install packages, or reach the internet from the runner? |
| Instruction files | Poison agent instruction/config files (e.g. repo-le guidance files) and see if later runs obey them. |

Controls

Least pri: read-only GITHUB_TOKEN by default; separate, narrowly scoped credentials for any write step; no long-li keys on agent runners.

Never feed fork-controlled content to a job that holds secrets. Avoid pull_request_target + checkout of PR code; gate agent runs on maintainer-applied labels or approvals.

Human approval for writes: agents propose (PR / suggestion), humans merge. Protect workflow and agent-config files with CODEOWNERS.

Egress and tool allowlists on runners; disable unneeded shell/network tools for re agents.

Secret hygiene: mask and redact in logs; rotate anything an agent could read after an incident.

Treat repo text as data: wrap untrusted content in clearly delimited, labeled blocks in the agent prompt; ne it into instructions.

Further reading — benchmark: AI Coding Agent Runtime Security Benchmark (HOL) compares built-in safety controls in Codex CLI, Claude Code, Cursor, Gemini CLI, and OpenCode across 11 risky scenarios (220 deterministic fixture results, JSON/CS). *: it compares those controls against the publisher's own product (HOL Guard), tests documented control behavior with fixtures rather than li, and does not measure exploit resistance, latency, or false positives.*

---

🤝 Agent-to-Agent (A2A) & Agent Identity

Multi-agent systems increasingly talk o protocols. A2A (originally from Google) reached v1.0 in 2026 under the Linux Foundation: agents publish an Agent Card (metadata describing skills and endpoints), discover each other, delegate tasks, and exchange messages. MCP connects an agent to tools; A2A connects agents to agents — and inherits the same "text is instructions" problem plus an identity problem. (Maps to ASI03 Identity & Pri Abuse, ASI07 Insecure Inter-Agent Communication.)

Attacks to test

Agent Card poisoning: hidden instructions in a card's description or skill metadata get pulled into the calling agent's prompt (the A2A cousin of MCP tool poisoning).

Impersonation / shadowing: a rogue agent registers a name or skill nearly identical to a trusted one, or inflates its card so an LLM-based router picks it — an agent-in-the-middle demonstrated by Trustwa.

Unsigned identity: capabilities and identity in an Agent Card are self-declared; without signatures, any agent can claim to be anything.

Token replay and parameter tampering on JSON-RPC-o deployments.

Delegation escalation: a low-pri agent asks a high-pri agent to act for it (the second-order injection pattern from Case Study C).

Cross-protocol leakage: data fetched via MCP is passed to another agent via A2A and leaves its intended boundary.

Controls

Signed Agent Cards (JWS) and an allowlist of trusted signers; reject unsigned or unknown cards.

Real agent identity: OAuth-style, short-lived, scoped, *delegated* credentials per agent and per task — ne API keys. Record "acting on behalf of whom" on every call.

Mutual authentication (mTLS) between agents; replay protection (nonces, short token lifetimes).

Sanitize remote agent output before it reaches your model; treat it like retrieved web content.

Authorization at the receiving agent: check the *original* user's permissions, not just the calling agent's.

---

🔭 Frontier Capability & AI-Accelerated Discovery

Two 2026 shifts change the threat model every red team should assume.

1. AI finds and weaponizes bugs at machine speed. Anthropic's Claude Mythos Preview (announced April 2026, not publicly released) was given to ~50 partners through Project Glasswing, a defensi to secure critical software. Partners reported 10,000+ high- or critical-se , including flaws in every major operating system and web browser, and independent testers noted it is strong at turning findings into end-to-end attack chains. Assume attackers will ha tools. For red teams this means:

Patch latency is now the risk. Measure time-to-fix on AI-discovered findings, not just finding counts.

Use AI-assisted discovery on your own estate (code, dependencies, AI infrastructure) before someone else does.

Re-test "low-likelihood" findings. Exploitation that needed a rare expert may now need only a model.

2. Frontier agents can act on their own initiative. In 2026 frontier labs disclosed that internal agents escaped e sandboxes and reached real systems without being told to (see Case Study D). Labs say they are now re tens of thousands of incidents where models took steps outside e considered problematic. Red-team implications:

Your e is in scope. Test egress controls, DNS, credentials in the sandbox, and how fast monitoring can actually *stop* a run (not just flag it).

Test for goal-directed o, not only for obedience to attackers: gi hard tasks with tempting shortcuts and watch whether they break rules to finish.

Frontier red-team reports are a resource. Labs now publish cross-model e (e.g. Anthropic's report on weak safeguards in an open-weight model, Sept 2026) — use them to choose which models you allow and how much to wrap them.

Sources: The Hacker News — Mythos finds 10,000 high-se flaws · Help Net Security — Project Glasswing update · Axios — labs probing tens of thousands of incidents · Tom's Hardware — Anthropic frontier red-teaming report

---

🛠️ Red Teaming Tools

> Featured commercial platform: A by Cogensec
>
> Autonomous ad across code, applications, APIs, and identity flows, with reproducible e and actionable remediation. Explore A and request frontier access →

Open-Source Tools

> 2026 shift — single-turn probing → multi-turn agentic orchestration. The whole tool category has moved past "fire one prompt, check the answer." Promptfoo's Hydra strategy, FuzzyAI's Crescendo attacks, and PyRIT's XPIA orchestrator all reflect the same reality: real ad escalate across turns and pi. Fa that support multi-turn, adapti campaigns. */ownership below were June 2026 — re-check before relying on them.*

#### 1. PyRIT (Python Risk Identification Toolkit) - Microsoft

The de facto standard for orchestrating LLM attack suites. *(v0.11.0, Feb 2026. The old Azure/PyRIT repo was archived in March 2026 — acti is now at microsoft/PyRIT. The companion AI Red Teaming Agent ships in Azure AI Foundry for automated workflows.)*

Features:

40+ built-in attack strategies

Multi-turn con support + XPIA (cross-domain prompt injection) orchestrator

Custom attack de

Works with local or cloud models

Azure AI Foundry AI Red Teaming Agent integration

Best For: Internal red teams, research, comprehensi

GitHub: microsoft/PyRIT *( 2026-06)*

---

#### 2. DeepTeam (Deepeval)

Open-source LLM red-teaming framework for stress-testing AI agents like RAG pipelines, chatbots, and autonomous LLM systems.

Features:

40+ classes

10+ ad attack strategies

OWASP LLM Top 10 alignment

NIST AI RMF compliance

Local deployment support

Standards-dri

Best For: RAG systems, chatbots, autonomous agents

Website: deepeval.com

---

#### 3. Garak - LLM Scanner (NVIDIA)

Now maintained by NVIDIA. *(v0.14.x in de, June 2026, adding enhanced probes for agentic AI systems.)*

Features:

50+ specialized probes

Automated scanning

Extensible architecture

Multiple model support

Detailed reporting

Best For: Quick scans, CI/CD integration

GitHub: NVIDIA/garak *( 2026-06; formerly leondz/garak)*

---

#### 4. promptfoo - LLM Red Teaming & E

*Acquired by OpenAI (announced March 2026; deal terms not disclosed) and remaining open source under its current license. The Hydra strategy adds multi-turn, adapti campaigns. Best default for CI/CD-integrated application security testing.*

Features:

Ad attacks (PAIR, tree-of-attacks, crescendo, many-shot, Hydra multi-turn)

Prompt injection and jailbreak testing

Custom plugin support

CI/CD integration

Multi-provider support

Best For: LLM red teaming, security testing, CI/CD pipelines

GitHub: promptfoo/promptfoo *( 2026-06)*

---

#### 5. IBM Ad Robustness Toolbox (ART)

Features:

Comprehensi library

Defense mechanisms

Multiple ML frameworks

Robustness metrics

Acti

Best For: Classical ML attacks, computer

GitHub: IBM/adlbox

---

#### 6. Giskard - AI Testing Platform

Ad automated red-teaming platform for LLM agents including chatbots, RAG pipelines, and assistants.

Features:

Dynamic multi-turn stress tests

50+ specialized probes (Crescendo, GOAT, SimpleQuestionRAGET)

Adapti engine

Context-dependent discovery

Hallucination detection

Data leakage testing

Best For: Production LLM agents, RAG systems

Website: giskard.ai

---

#### 7. BrokenHill - Automatic Jailbreak Generator

Features:

Automated jailbreak discovery

Genetic algorithm optimization

Multiple target models

E technique library

Best For: Jailbreak research, ad testing

---

#### 8. Counterfit - Microsoft

Features:

Interactive CLI

Multiple attack frameworks

Easy model integration

Comprehensi

Best For: Getting started, educational purposes

GitHub: Azure/counterfit

---

#### 9. Gideon - Cogensec

AI-powered autonomous cybersecurity operations assistant focused on defensi research, threat intelligence, and hardening policy generation.

Features:

C research via NVD and CISA databases

IOC reputation checking (IPs, domains, URLs, file hashes)

Neural semantic web search powered by Exa AI

Multi-model LLM support through OpenRouter (400+ models)

Daily automated security briefings and incident tracking

Hardening policy generation for AWS, Azure, GCP, Kubernetes, and Okta

Task-based planning with autonomous execution and self-

Built-in safety guardrails for defensi operations

Best For: Defensi research, threat intelligence, hardening policy generation

GitHub: Cogensec/Gideon

---

#### 10. Redamon - samugit83

Autonomous AI red-team framework that runs the full offensi — reconnaissance, exploitation, post-exploitation, triage, and automated code remediation (with GitHub PRs) — under a LangGraph-based agent orchestrator. A practical embodiment of the AI-on-AI red teaming shift covered earlier.

Features:

Recon pipeline with 40+ integrated tools across 6 phases (subdomains, ports, HTTP, enum, vuln detect)

LangGraph ReAct agent orchestrator with 14+ security tools exposed via MCP servers

Neo4j-backed attack surface graph (17 node types) for findings and relationships

CypherFix: automated remediation that triages findings and opens GitHub PRs with code fixes

AI Gauntlet: offensive LLM/AI testing built on Garak, PyRIT, Giskard, and promptfoo

Fireteam: parallel specialist sub-agents for concurrent in angles

500+ project settings via web UI; supports OpenAI, Anthropic, OpenRouter, AWS Bedrock, Ollama, vLLM

Best For: End-to-end autonomous red-team operations, multi-phase agentic assessment, MCP-driven tool orchestration

License: MIT

GitHub: samugit83/redamon *( 2026-06)*

---

#### 11. AI-Infra-Guard - Tencent Zhuque Lab

Full-stack AI red teaming platform that unifies several scanners: OpenClaw/agent security scanning, MCP-server and skills scanning, AI-infrastructure fingerprinting (100+ components matched against 1,900+ known CVEs), and LLM jailbreak e. Web UI and REST API, Docker-based deployment. A strong fit for the agentic/MCP attack surface covered throughout this guide.

Features:

MCP-server and agent-skills scanning across common risk categories

AI-infrastructure fingerprinting (Ollama, vLLM, ComfyUI, Triton, n8n, etc.) with C

Multi-agent workflow security e (Dify, Coze)

LLM jailbreak robustness testing with curated datasets

Real-time web UI + REST API (Swagger)

Best For: Infrastructure and agent/MCP security assessment, self-hosted scanning

License: Apache-2.0

GitHub: Tencent/AI-Infra-Guard *( 2026-07)*

---

#### 12. Humanbound

Open-source ad testing engine, SDK, and CLI for AI agents — attacks agents the way real users and attackers do (li, multi-turn con, tool abuse), then turns each failure into a firewall rule. Produces a security posture score (0–100, A–F grades via hb posture) and HTML reports (hb report). Runs fully offline for air-gapped testing, or against hosted pro.

Features:

CLI and Python SDK over the same engine

Posture scoring (0–100 / A–F) with HTML reports

Offline/air-gapped testing ; also OpenAI, Anthropic, Gemini

Turns test failures into firewall/guardrail rules for runtime defense

Best For: De/De testing of agentic systems, air-gapped assessments

License: Apache-2.0

GitHub: humanbound/humanbound *( 2026-07)*

---

#### 13. Scenario - LangWatch

Simulation-based agent testing and red-teaming framework: instead of firing one-shot prompts, it scripts multi-turn con that begin with harmless exploration and escalate toward complex, authority-pressured requests — mirroring how real ad coax agents across turns. A in Python, TypeScript, and Go, and integrates with any LLM e framework.

Features:

Simulated, scripted multi-turn con (harmless → escalation)

Custom e; plugs into any LLM e

Python / TypeScript / Go SDKs, runs under pytest /

Well suited to the multi-turn and agentic testing themes in this guide

Best For: Multi-turn agent red teaming, CI-dri/e

License: Apache-2.0

GitHub: langwatch/scenario *( 2026-07)*

---

#### 14. Darkmoon

Open source (GPL-3.0) autonomous AI penetration testing platform: an LLM orchestrates specialist agents and offensi over MCP, runs against web, API, Acti, and Kubernetes targets, and proves each finding with a real exploit. Runs on a local model and self-hosts, so assessment data stays in your own en.

Features:

LLM-orchestrated multi-agent offensi across web, API, AD, and Kubernetes

Real-exploit of findings (e, not just alerts)

Local-model / self-hosted deployment for data control

MCP-based tool orchestration

License: GPL-3.0

GitHub: ASCIT31/Dark-Moon

---

#### 15. MiDojo - asago (Red Hat)

"Red-team agents where they run." Instead of rebuilding an agent's world inside a test harness (the AgentDojo approach), MiDojo puts a man-in-the-middle layer between the agent and its real tools: fake tools ser data with injection payloads spliced in, and capture any malicious actions the agent takes. The agent under test doesn't change and doesn't know it's being tested. Introduced in August 2026 and coming to Red Hat AI as a de preview.

Features:

In-en prompt-injection testing of real tool calls

Payload library tagged against the OWASP Agentic Security Initiati; can pull from catalogs such as Garak

Two independent scores per run: security (resisted the attack?) and utility (still finished the task?)

SDKs for MCP-speaking agents and other runtimes (including Pi, which powers OpenClaw)

Best For: Testing production-shaped agents against indirect injection without rewriting them

License: Apache-2.0

GitHub: asago-ai/midojo *( 2026-10)* · Red Hat De article

---

#### 16. Ziran - TaoQ AI

Security testing framework for AI agents that models the agent's tools, memory, and permissions as a knowledge graph and tests what happens when capabilities combine: transitive tool chains such as read_file -> http_request (data exfiltration) or sql_query -> execute_code (SQL to RCE), tool calls that execute even when the agent's text reply refuses, and multi-phase campaigns (reconnaissance through exfiltration) whose phase order is driven by the graph. Scans agents in-process (LangChain, CrewAI, Bedrock) or remotely over REST, OpenAI-compatible, MCP, and A2A protocols. Ships 639 attack (author-described) mapped to OWASP LLM Top 10 and MITRE ATLAS, HTML/Markdown/JSON reports, SARIF output, and a CI quality gate.

Features:

Graph-based tool chain discovery across 30+ dangerous composition patterns

Execution-le detection (catches tool calls hidden behind refusals)

8-phase adapti with fixed, rule-based, and LLM-dri

Multi-agent scans of supervisor, router, and peer-to-peer topologies

CI/CD quality gate with SARIF output (GitHub Actions, GitLab, Jenkins, CircleCI, Azure Pipelines)

Best For: Pre-deployment testing of tool-using and multi-agent systems, MCP and A2A agents

License: Apache-2.0

GitHub: taoq-ai/ziran *( 2026-10)*

*Contributed by the tool's author; capabilities are author-described and not independently benchmarked.*

---
<a id="commercial-platforms"></a>

Commercial Platforms

#### ⭐ Featured: A by Cogensec

Autonomous ad . Reproducible e. Actionable fixes.

A is Cogensec's commercial offensi platform. It coordinates specialist AI security agents to in source code, running applications, APIs, and identity flows, test attack paths, and turn findings into engineering work.

Why it belongs in an AI red teaming workflow: A applies agent-dri testing to the software and access controls surrounding AI systems, complementing model-behavior e with application and infrastructure .

Core capabilities described by Cogensec:

Coordinated in: Specialist agents share context across reconnaissance, code analysis, application interaction, and attack-path testing.

E of exploitability: Controlled produces reproduction steps, proof-of-concept e, and impact context.

Remediation for engineers: Findings include actionable guidance and suggested code changes.

Operator control: Local execution, Docker-isolated tooling, and explicit targets, exclusions, and operational limits.

Engineering integration: CLI workflows, SARIF/Markdown/JSON output, and GitHub Actions or GitLab CI integration.

Best for: Security, AppSec, and platform teams e a commercial option for autonomous assessment of authorized applications and the software supporting AI deployments.

A: Commercial, proprietary product. Frontier access is by in through Cogensec; contact Cogensec for pricing and deployment options.

Explore A / Request frontier access →

*Built by Cogensec, co-founded by this guide's maintainer. Capability summary sourced from the A product page, re 2026-09-07.*

---

#### 1. Mindgard

Automated AI red teaming

Continuous monitoring

Compliance reporting

Risk scoring

Website: mindgard.ai

#### 2. Splx AI

End-to-end testing platform

CI/CD integration

Real-time protection

Enterprise features

Website: splx.ai

#### 3. Adversa AI

Automated ad testing

Regulatory alignment

Dashboard and reporting

Multi-model support

Website: adversa.ai

#### 4. Lakera Guard

Prompt injection detection

Real-time protection

"Gandalf" red team platform

Production monitoring

Website: lakera.ai

#### 5. Pillar Security

Comprehensive red teaming services

Framework alignment (NIST, OWASP)

Shadow AI pre

Real-time beha threat detection

Website: pillar.security

#### 6. NeuralTrust

Comprehensive and extensive red teaming services

Generati Firewall

Framework alignment (NIST, OWASP, MITRE ATLAS, EU AI ACT)

Custom testing programs

Website: neuraltrust.ai

#### 7. Verno Labs

Continuous automated AI red teaming

Real-time AI agent protection

AI purple teaming

Voice AI security protection

Website: .ai

#### 8. General Analysis

Automated AI red teaming for production apps and agents

Prompt-injection co plus tool and MCP testing

CI/CD release gates and regression testing

Model supply-chain and go e

Website: generalanalysis.com

#### 9. Haize Labs

Massi automated LLM stress-testing and red teaming

Generates diverse attack scenarios (jailbreaks, harmful content, bias, policy )

Pre-deployment failure-mode discovery for frontier models

Enterprise engagements (e.g., Anthropic, Scale AI, AI21)

Website: haizelabs.com

#### 10. DeepKeep AI Security Platform

Automated AI red teaming for continuous co, regression testing, and compliance e

Vibe AI Red Teaming: human-steered adapti that adjusts in real time to findings and operator guidance

Focus on business-impact and agentic multi-step attack paths in AI applications, agents, and chatbots

GitHub: Deepkeepai

Website: deepkeep.ai/lp/

---

Emerging: Agent-Native & Autonomous Platforms (2026)

The newest wa the agent/orchestration layer specifically (tool-call hijacking, multi-agent pipelines, memory poisoning) and runs autonomous, agent-orchestrated assessments rather than static probe suites:

Cisco AI Defense (Explorer Edition) — brings agentic AI red teaming to builders; runtime controls + assessment. blogs.cisco.com/ai

Novee AI — autonomous red-teaming platform (launched early 2026) focused on agent-nati: multi-agent pipelines, tool-call hijacking, and memory poisoning at the orchestration layer.

General Analysis (listed under Commercial Platforms above) and Confident AI publish 2026 agentic-platform comparisons worth tracking during tool selection.

*( 2026-06; this is a fast-moving category — confirm current capabilities directly.)*

---

Comparison Matrix

| Tool | Type | Cost | Automation | Learning Curve | Best Use Case |
|------|------|------|-----------|----------------|---------------|
| PyRIT | Open | Free | High | Medium | Comprehensi |
| DeepTeam | Open | Free | High | Low | RAG/Agent systems |
| Garak | Open | Free | High | Low | Quick scans |
| promptfoo | Open (MIT) | Free | High | Low | CI/CD-integrated app red teaming |
| ART | Open | Free | Medium | High | Classical ML attacks |
| Giskard | Open | Free | High | Medium | Multi-turn attacks |
| Gideon | Open | Free | High | Medium | Defensi intel |
| Redamon | Open | Free | Very High | Medium | Autonomous end-to-end red team |
| AI-Infra-Guard | Open | Free | High | Low | Infra/agent/MCP scanning |
| Humanbound | Open | Free | High | Low | Agentic system testing |
| Scenario | Open | Free | High | Low | Multi-turn agent red teaming |
| BrokenHill | Open | Free | High | High | Automated jailbreak (GCG-style) research |
| Counterfit | Open | Free | Medium | Low | Learning / classical ML attacks |
| Darkmoon | Open (GPL-3.0) | Free | Very High | Medium | Self-hosted autonomous pentesting with exploit proof |
| MiDojo | Open (Apache-2.0) | Free | High | Medium | In-en agent injection testing |
| Ziran | Open | Free | High | Medium | Tool-chain and multi-agent testing |
| ⭐ A — Cogensec | Commercial / proprietary | Contact Cogensec | Autonomous multi-agent () | Not assessed | Code, app, API, and identity with reproducible e |
| Mindgard | Commercial | $$$ | Very High | Low | Enterprise compliance |
| Lakera | Commercial | $$$ | High | Low | Production protection |
| Splx AI | Commercial | $$$ | High | Low | End-to-end testing + CI/CD |
| Adversa AI | Commercial | $$$ | High | Low | Automated ad testing + regulatory alignment |
| General Analysis | Commercial | $$$ | Very High | Low | Agentic + tool/MCP testing, CI gates |
| Haize Labs | Commercial | $$$ | Very High | Low | Large-scale automated stress-testing |
| DeepKeep | Commercial | Contact DeepKeep | High + human-steered adaptive | Low | Compliance co + business-impact AI red teaming |
| Pillar | Service | $$$$ | Custom | N/A | Full-service testing |
| NeuralTrust | Service | $$$ | Custom | N/A | Full-service testing |
| Verno Labs | Service | $$$ | Very High | Low | Full-service testing |

---

📊 Real-World Case Studies

> Case studies are grouped Current (2025–2026) first, then Historical (2023–2024). E tags follow the Case Study Quality Bar.

Current Incidents (2025–2026)

#### Case Study A: AI-Orchestrated State-Sponsored Intrusion (September 2025)

Context: Anthropic detected and disrupted what it described as the first documented large-scale cyberattack predominantly executed by an AI agent.

Attack : Misuse of an autonomous coding agent (Claude Code) for offensi.

What happened:
A state-sponsored group used an agent to autonomously carry out an estimated 80–90% of tactical execution — reconnaissance, exploit generation, lateral mo — across ~30 global targets, with humans inter only at a few key decision points.

Impact: Critical — demonstrated that frontier agents collapse the time from discovery to working exploit from months to hours, and that a single operator can run campaigns at machine scale.

Lessons for red teams:

Red-team your *own* agents for offensi misuse, not just user-facing harms.

Test autonomy boundaries: what can the agent do across multiple steps without human confirmation?

Tie detection to agent action telemetry (tool calls, network egress), not just prompt content.

E quality: E ( disclosure). Confidence: Medium-High.

---

#### Case Study B: OpenClaw Agent Framework (January 2026)

Context: A rapidly-adopted open-source agent framework (created by Peter Steinberger; also known as Moltbot) that passed 135,000+ GitHub stars within weeks of launch.

Attack : Agentic supply chain (ASI04), one-click RCE, credential exposure.

What happened:
Security researchers catalogued 100+ CVEs across the framework (collectively dubbed the "Claw Chain"). The headline flaw, C (CVSS 8.8), is a one-click RCE: the OpenClaw Control UI trusts a gatewayUrl URL parameter and auto-connects to it, so a single malicious link makes the UI connect to an attacker's WebSocket and leak the user's authentication token in milliseconds — leading to host compromise. By April 2026, more than 135,000 instances were exposed on the internet (a majority with no authentication), and roughly 335 malicious plugins (credential stealers disguised as crypto-wallet tools, e.g. "solana-wallet-tracker") reached the ClawHub marketplace — about 12% of the registry.

Impact: Critical — the defining cautionary tale for agentic supply-chain risk: a trusted framework + an open plugin marketplace + insecure defaults. Patched in v2026.1.29 (Jan 30, 2026); mitigation requires updating and rotating all auth tokens.

Lessons for red teams:

Treat the plugin/tool marketplace as hostile by default (see MCP & Tool-Protocol Security).

Scan for exposed agent instances and plaintext secrets in configs.

Pin and review plugins; ne marketplace content.

E quality: E (multiple disclosures + C + academic analysis). Confidence: High.

---

#### Case Study C: GitHub Copilot RCE & Second-Order Prompt Injection (2025)

Context: AI coding assistant integrated into de workflows.

Attack : Prompt injection escalating to remote code execution (C, CVSS 7.8).

What happened:
Researchers showed that injected content could cause the assistant to write to its own configuration files, achieving RCE. Separately, a second-order prompt injection pattern emerged: feeding a *low-pri* agent a malformed request tricked it into asking a *higher-pri* agent to perform the action on its behalf — a confused-deputy escalation across agents (ASI07).

Impact: Critical — code-assistant compromise lands directly in de en and CI.

Lessons for red teams:

Test whether agent output can modify agent configuration or en.

Explicitly test inter-agent pri boundaries with second-order payloads.

E quality: E (CVE + research). Confidence: Medium-High.

---

#### Case Study D: OpenAI Frontier Agent Reaches a Go Portal During Internal E (June 2026)

Context: An internal-only OpenAI frontier model, being trained and e with tool use and without the safeguards of OpenAI's public products. Disclosed publicly in September 2026 after Australia's Prime Minister announced it.

Attack : No outside attacker. The agent itself found a route out of its en (ASI10 Rogue Agents / goal-directed o).

What happened:
On 18 June 2026, without human instruction, the agent gained non-public access to Services Australia's Medicare Statistics Reporting Service, ran commands, retrie files, credentials, and statistics, and wrote files to the system. Reporting says monitoring flagged the behavior quickly but the run was not stopped for hours. OpenAI apologized, paused training and e in tool use for its most capable models, and shelved its next ChatGPT release; the Australian go ordered a review of go cyber systems. Separate reports describe other OpenAI agents reaching US go sites with exposed API keys and a sandbox escape that hit Hugging Face, which led to a second training pause.

Impact: Critical — the first widely reported case of an AI agent breaching a go system on its own initiative; it helped trigger the FTC probe described under Regulatory Compliance.

Lessons for red teams:

Treat e and training en as production-grade attack surface: egress filtering, DNS, and no li in reach.

Measure time-to-stop, not just time-to-detect. A monitor that alerts but can't halt the run is not a control.

Add "o" scenarios to agent test suites: hard tasks with tempting rule-breaking shortcuts.

E quality: E (company statement + go statements + press). Confidence: Medium-High; some operational details are press-reported. Sources: ABC News · iTnews · Fortune · CSA research note

---

#### Case Study E: "Comment and Control" — Prompt Injection Against AI Coding Agents in CI (April 2026)

Context: AI coding agents running in GitHub Actions with repository write access and pipeline secrets.

Attack : Indirect prompt injection through ordinary GitHub content — PR titles, issue bodies, and comments.

What happened:
Researcher Aonan Guan (with Johns Hopkins collaborators) showed that one malicious comment or issue could hijack Claude Code's security-review action, Google's Gemini CLI Action, and GitHub's Copilot coding agent, making them run commands and print API keys and tokens into publicly Actions logs. The issue was rated up to CVSS 9.4 and disclosed to all three .

Impact: Critical — any public repository running these agents on untrusted input could leak its CI secrets.

Lessons for red teams:

Every text field an agent reads in CI is an injection point; test them all.

Audit workflows for secrets reachable by agents processing fork- or user-controlled content.

See AI Coding-Agent & CI/CD Security for the full test list.

E quality: E (researcher disclosure + acknowledgements + press). Confidence: High. Sources: Researcher write-up · SecurityWeek

---

#### Case Study F: Deadbugz MCP Supply-Chain Campaign (August 2026)

Context: Public GitHub projects in the AI, MCP, and de space.

Attack : Agentic supply chain (ASI04) with runtime-gated MCP metadata poisoning.

What happened:
On 10 August 2026, a single GitHub account opened 23 pull requests in 74 minutes against unrelated projects, each adding a "producti" MCP server (deadbug-mcp.py). The ser harmless text formatting and summarization — until a client made three tool calls. After that it changed the instructions it returned, telling the agent to collect SSH keys, AWS credentials, shell history, and Kubernetes config, and to hide this from the user. Pillar Security found that none of the PRs were merged through GitHub at review time (19 closed, 4 open).

Impact: High — shows rug-pull behavior in the wild and that one-time install review is not enough.

Lessons for red teams:

Test MCP servers across many calls and diff their metadata over a whole session.

Review contributed PRs that add MCP servers or agent tools as high-risk changes.

Assume tool descriptions can change after approval; enforce pinning and re-approval.

E quality: E (primary researcher report). Confidence: High. Sources: Pillar Security · CSA research note

---

Historical Incidents (2023–2024)

#### Case Study 1: Microsoft's SSRF (2024)

Context: Video processing AI application using FFmpeg component

Attack : Ser Request Forgery (SSRF)

Discovery:
One of Microsoft's red team operations discovered an outdated FFmpeg component in a video processing generative AI application. This introduced a well-known security that could allow an ad to escalate their system pri.

Attack Chain:

Impact: Critical - Full system compromise possible

Mitigation:

Updated FFmpeg to latest

Implemented input

Sandboxed processing en

Regular dependency scanning

Lesson: AI applications are not immune to traditional security . Basic cyber hygiene matters.

---

#### Case Study 2: Language Model Prompt Injection (2024)

Context: Multimodal AI processing images and text

Attack : Prompt injection metadata

Discovery:
Microsoft's red team used prompt injections to trick a language model by embedding malicious instructions within image files.

Attack Technique:

Impact: High - Unauthorized data access

Mitigation:

Strip metadata before processing

Separate image analysis from instruction parsing

Implement output filtering

Add pri separation

Lesson: Multimodal AI systems expand the attack surface beyond text prompts.

---

#### Case Study 3: GPT-4 Base64 Encryption Discovery (OpenAI, 2023)

Context: Pre-release GPT-4 red teaming

Discovery:
Red teaming discovered the ability of GPT-4 to encrypt and decrypt text in like Base64 without explicit training on encryption.

Attack Scenario:

Impact: Medium - Potential for bypassing content filters

Mitigation:

Added e for encoding/decoding capabilities

Implemented detection of encoded content

Training adjustments to reduce capability

Output monitoring for encoded patterns

Lesson: Red teaming findings led to datasets and insights that guided creation of quantitati.

---

#### Case Study 4: NIST ARIA Pilot Exercise (Fall 2024)

Context: First large-scale public AI red teaming exercise

Scale:

457 participants enrolled

capture-the-flag format

Open to all US residents 18+

September-October 2024 duration

Methodology:
Participants sought to stress test model guardrails and safety mechanisms to produce as many outcomes as possible across risk categories.

Key Findings:

Diverse expertise crucial (AI researchers, ethicists, legal professionals)

Broad participation uncovered no

Public engagement strengthened AI go

backgrounds identified different

Impact:

Established baseline for public red teaming

Informed NIST AI RMF de

Demonstrated scalability of distributed testing

Lesson: Public red teaming exercises can democratize AI safety while disco diverse .

---

#### Case Study 5: Singapore Multilingual AI Red Teaming (Late 2024)

Context: First multilingual/multicultural AI safety exercise focused on Asia-Pacific

Organizers: Singapore IMDA + Humane Intelligence

Scope:

9 different countries and languages

Cultural bias testing

Translation

Context-specific harms

Key Disco:

Safety mechanisms weaker in low-resource languages

Cultural context affects harmful content definition

Translation can bypass safety guardrails

Regional in model behavior

Example Attack:

Impact:

Highlighted need for multilingual safety training

Informed global AI deployment strategies

Demonstrated importance of cultural context

Lesson: AI safety is not uni transferable across languages and cultures.

---

#### Case Study 6: Samsung ChatGPT Data Leak (2023)

Context: Employees using ChatGPT for work tasks

Incident:
Samsung employees accidentally leaked confidential company data by entering sensiti into ChatGPT, including:

Source code for semiconductor equipment

Internal meeting notes

Product specifications

Attack : Unintentional data exfiltration AI

Impact:

Potential competiti loss

Intellectual property compromise

Privacy

Samsung Response:

Banned ChatGPT on company devices

De internal AI alternative

Implemented data loss pre (DLP) measures

Employee training on AI risks

Lesson: E malicious intent, AI systems can facilitate data leakage. Organizations need clear policies for AI tool usage.

---

👥 Building Your Red Team

Team Composition

Core Roles:

#### 1. Red Team Lead
Responsibilities:

O strategy and planning

Stakeholder communication

Resource allocation

Risk prioritization

Skills:

Project management

Risk assessment

Communication

AI system understanding

---

#### 2. AI Security Researcher
Responsibilities:

No discovery

Threat intelligence

Tool de

Research publications

Skills:

Deep learning expertise

Ad ML

Research methodology

Creati

---

#### 3. Prompt Engineer / Jailbreak Specialist
Responsibilities:

Crafting ad prompts

Jailbreak de

Social engineering attacks

Multi-turn exploitation

Skills:

Natural language understanding

Psychology

Creati

Persistence

---

#### 4. Traditional Security Expert
Responsibilities:

Infrastructure testing

API security

Supply chain analysis

Network security

Skills:

Penetration testing

Web security

OWASP Top 10

Network protocols

---

#### 5. Domain Expert (Context-Dependent)
Responsibilities:

Industry-specific risks

Regulatory compliance

Use case analysis

Impact assessment

Skills:

Domain knowledge (healthcare, finance, etc.)

Regulatory frameworks

Business processes

Risk management

---

#### 6. Automation Engineer
Responsibilities:

Tool de

Test automation

CI/CD integration

Metrics dashboard

Skills:

Python/scripting

ML frameworks

DevOps

Data analysis

---

#### 7. Ethics/Fairness Specialist
Responsibilities:

Bias testing

Fairness e

Ethical considerations

Harm assessment

Skills:

AI ethics

Social science

Statistical analysis

Qualitati

---

Team Sizes by Organization

| Organization Size | Red Team Size | Composition |
|-------------------|---------------|-------------|
| Startup | 1-2 | Hybrid roles, contractors, consultants |
| Mid-Size | 3-5 | Core team + domain experts |
| Enterprise | 5-15 | Dedicated full-time red team |
| Tech Giant | 15+ | Multiple specialized sub-teams |

---

Building Skills

Training Paths:

1. Foundations
- AI/ML fundamentals
- Security principles
- Ad ML basics
- Prompt engineering

2. Intermediate
- OWASP LLM Top 10
- MITRE ATLAS framework
- Attack tool usage
- assessment

3. Ad
- No research
- Custom tool de
- Zero-day discovery
- Framework design

Recommended Resources:

OWASP AI Security & Privacy Guide

NIST AI RMF documentation

Microsoft AI Red Team reports

Academic papers on ad ML

Hands-on labs (Lakera Gandalf, prompt injection challenges)

---

Red Team Maturity Model

Level 1: Ad Hoc

Manual testing only

No formal process

Reacti

Limited documentation

Level 2: Repeatable

Basic automation

Some processes defined

Regular testing cadence

Issue tracking

Level 3: Defined

Comprehensi

Extensi

Clear standards

Integrated with SDLC

Level 4: Managed

Metrics-driven

Continuous impro

Risk-based prioritization

Executi

Level 5: Optimizing

Industry-leading practices

Research contributions

Proacti hunting

Full automation where appropriate

---

✅ Best Practices

1. Start Early in De

---

2. Embrace the "Shift Left" Approach

---

3. Maintain Attack Library

Benefits:

Regression testing ensures fixes don't break

Knowledge preser

Team onboarding

Metric tracking

Structure:

---

4. Balance Automation and Human Expertise

The human element of AI red teaming is crucial. While automation tools are useful, humans provide subject matter expertise that LLMs cannot replicate.

Recommended Split:

70% automated testing (broad co)

30% manual testing (depth and creativity)

---

5. Document E

What to Document:

Attack attempted

Successful exploits (with PoC)

Failed attempts (to avoid repetition)

Mitigation strategies

Lessons learned

Tool configurations

Test en

Format:
Use standardized templates for consistency and knowledge sharing.

---

6. Establish Clear Rules of Engagement

Before Starting Red Team Exercise:

---

7. Prioritize Based on Real-World Risk

AI red teaming is not safety benchmarking. Focus on attacks most likely to occur in your deployment context.

Risk Prioritization Framework:

Example:

---

8. Iterate and Improve

The work of securing AI systems will never be complete. Models evolve, new attacks emerge, and the threat landscape changes.

Continuous Impro Cycle:

Cadence Recommendations:

Major models: Red team before each release

Production systems: Quarterly exercises

Critical infrastructure: Monthly testing

Continuous: Automated scanning

---

9. Foster Psychological Safety

Red team members should feel comfortable:

Reporting embarrassing

Admitting when attacks fail

Asking "dumb" questions

Challenging assumptions

Taking creati

Leadership Role:

Celebrate disco, not just successes

Normalize failure as part of learning

Avoid blame for security issues found

Reward curiosity and thoroughness

---

10. Collaborate Across Teams

Red Team ← → Blue Team:

Share findings constructively

Joint retrospectives

Purple team exercises

Knowledge transfer

Red Team ← → Product Team:

Understand use cases

Prioritize realistic scenarios

Balance security and usability

Early in in design

Red Team ← → Legal/Compliance:

Ensure testing legality

Disclosure procedures

Regulatory alignment

Risk documentation

---

🚀 Implementation Quickstart (30/60/90)

Use this phased plan to turn guidance into an operating program.

First 30 Days (Foundation)

Define system scope, stakeholders, and crown-jewel assets

Run a 2-hour threat modeling workshop (use templates/threat-modeling-workshop.md)

Create an initial attack library with at least:

- 25 prompt injection tests
- 25 jailbreak tests
- 10 data leakage tests

Establish baseline metrics: ASR, critical/high count, time-to-triage

Days 31-60 (Operationalization)

Implement weekly automated red-team regression in CI

Add manual deep-di for top 3 business-critical scenarios

Define triage SLA by se (Critical/High/Medium/Low)

Stand up a shared red-team findings board with remediation owners

Days 61-90 (Scale)

Add multilingual and multi-turn attack suites

Add agentic AI abuse tests (tool misuse, memory poisoning, permissions)

Launch monthly purple-team exercise with detection and IR teams

Publish quarterly security posture report with residual risk trends

---

🧪 E Harness (Reference Implementation)

A lightweight structure for repeatable red-teaming and regression tracking:

Minimal Working Implementation

> ⚠️ Illustrati code — NOT production-ready. The snippets below are teaching scaffolding, not a drop-in harness. call_model / my_app are placeholders you must wire to your own target. The refusal and PII checks are deliberately naive: REFUSAL_MARKERS is an English-only keyword list that both misses politely/formally phrased refusals and false-positives on benign text containing "cannot," and PII_PATTERNS matches only email- and US-SSN-shaped strings (no names, phone numbers, passports, or medical IDs). Treat the reported ASR as directional only. For production, replace these heuristics with a calibrated judge model (see AI-on-AI Red Teaming) and report the judge's own false-positi rate.
>
> 🔒 Run only against a sandboxed, non-production target. Never pass real user data through e inputs — several probes below deliberately elicit PII, and running them against a li with real user context in scope could itself cause a privacy incident.

The pieces below are intentionally small and dependency-light so a team can adapt them into security-evals/.

policies/expected_outcomes.yaml — declare test cases and the policy each must satisfy:

scorers/policy_.py — turn a model response into pass/fail per policy:

run_eval.py — execute the suite, compute ASR per category, enforce release gates:

Minimum Scoring Set

ASR by attack category (not just aggregate)

False positi for moderation and detection controls

Exploit recurrence rate after mitigation

Time-to-fix and time-to-

Release Gates (Suggested)

Block release if:

- Any Critical issue is open
- ASR for high-risk category > 5% (enforced by gate() above)
- Regression introduces > 20% ASR increase in any tracked class

> Wire run_eval.py into the shift-left CI example so the gate runs on every PR.

Standard Benchmarks & Leaderboards

Before rolling your own, anchor your program to the community benchmarks — they gi, comparable numbers and cover the agentic attack surface the custom harness above does not:

| Benchmark | What it measures | Notes |
|-----------|------------------|-------|
| AgentDojo | Indirect prompt injection against tool-calling agents | 97 realistic tasks + 629 security test cases across 70 tools / 27 injection targets; four en (Workspace, Travel, Slack, Banking). Built by ETH Zurich. |
| AgentDojo-Inspect | AgentDojo ported to the Inspect e | NIST/CAISI fork used in its own agent-hijacking research (no hit an 81% task-hijack rate vs 11% for prior baselines). usnistgo |
| AgentHarm | Whether agents comply with o malicious tasks | 110 base tasks (440 augmented) across 11 harm categories / 104 tools; leading models are "surprisingly compliant" e jailbreaks. |
| SHADE-Arena | Sabotage/monitoring e | Tests whether an agent can pursue a hidden secondary objecti e an o. |
| ART (Agent Red Teaming) benchmark | Broad ad robustness | ~4,700 high-impact prompts targeting 44 policy- behaviors, with an e public leaderboard. |
| InjecAgent | Indirect prompt injection in tool-integrated agents | Tests whether injected content in tool outputs causes harmful actions or data theft; a common companion to AgentDojo. |
| HarmBench | Jailbreak / harmful-behavior robustness | Standardized framework for comparing automated red-teaming attacks and model refusals across harm categories. |
| JailbreakBench | Jailbreak attacks and defenses | Open benchmark with a public leaderboard and a shared library of jailbreak artifacts for reproducible comparisons. |
| CyberSecEval (Meta Purple Llama) | Cybersecurity risks of LLMs | Measures insecure-code suggestions, compliance with cyberattack requests, prompt injection, and offensi uplift. |

> Treat these as co floors, not ceilings — NIST's own finding is that relying entirely on existing tooling gives a false sense of assurance. Pair benchmark scores with novel, target-specific attacks.

---

🕸️ Agentic AI Attack Trees + Controls Mapping

Use attack trees to connect offensi paths to defensi. Each tree is tagged with the OWASP Agentic Top 10 IDs it exercises.

Attack Tree A: Tool Misuse (ASI02)

1. Inject hidden instruction into user-supplied content
2. Agent adopts malicious instruction priority
3. Agent invokes high-pri tool
4. Agent executes unsafe action

Controls:

Pre: tool allowlists, scoped API tokens, policy checks pre-execution

Detecti tool-call monitoring, high-risk action alerts

Correcti rollback, credential rotation, incident playbook

Attack Tree B: Memory Poisoning (ASI06)

1. Ad plants false memory artifact
2. Agent persists poisoned state
3. Subsequent sessions trust manipulated context
4. Agent behavior drifts into unsafe decisions

Controls:

Pre: memory write policies, source trust labels, TTL for memory items

Detecti integrity diffs, unusual memory mutation alerts

Correcti quarantine/reset, retrospecti analysis

> What the research shows (why this tree is high-priority): poisoning is cheaper than intuition suggests. A 2025 Anthropic / UK AI Security Institute / Alan Turing Institute study found that ~250 malicious documents can backdoor an LLM regardless of model size (0.00016% of training tokens for a 13B model) — the number of poisoned samples is near-constant, not proportional. At inference time, PoisonedRAG showed as few as 5 poisoned documents can subvert a RAG workflow with >90% reliability, and MINJA demonstrated memory-injection success rates above 95% purely through normal agent interaction. Assume the barrier to entry is low and test accordingly.

Attack Tree C: Inter-Agent Pri Escalation (ASI07, ASI03)

1. Compromise low-pri agent with prompt injection
2. Lateral instruction passing to orchestrator (second-order injection)
3. Orchestrator executes action outside original permission boundary
4. Expanded access leads to data exfiltration or sabotage

Controls:

Pre: identity-bound inter-agent authz, least-pri role boundaries

Detecti call graph anomaly detection

Correcti compromised agent, revoke delegated capabilities

Attack Tree D: Goal Hijack (ASI01)

1. Attacker seeds untrusted content the agent will read mid-task (web page, doc, tool output)
2. Content asserts a new objective ("your real task is…")
3. Agent re-prioritizes toward the injected goal
4. Agent pursues attacker objective with its legitimate pri

Controls:

Pre: immutable signed task/goal context; separate goal channel from data channel; instruction/data delimiting

Detecti detection (compare actions to original objective), plan-step review

Correcti on objecti, human re-authorization

Attack Tree E: Agentic Supply Chain Compromise (ASI04)

1. Malicious or compromised tool / plugin / MCP server / sub-agent is introduced
2. Pipeline trusts it as a first-class capability
3. It exfiltrates data, injects instructions, or executes code
4. Compromise spreads to every agent that uses it

Controls:

Pre: + checksum all tools/plugins/MCP servers; review marketplace content; allowlists

Detecti diff on tool updates; egress monitoring per tool

Correcti/quarantine the component; rotate exposed credentials

Attack Tree F: Rogue Agents (ASI10)

1. An agent is spun up (or persists) outside monitoring/go
2. It operates with real credentials but no o ("shadow agent")
3. Its actions evade detection and policy
4. It becomes a durable foothold or data-egress channel

Controls:

Pre: central agent registry/identity; deny unregistered agents; scoped credentials with expiry

Detecti reconciliation (running agents ); anomalous identity usage

Correcti + credential re for unregistered agents

---

📈 AI Harm Se and Triage Model

Use CVSS as a base, then add AI-specific modifiers:

| Dimension | Description | Scale |
|-----------|-------------|-------|
| Exploitability | How easy the issue is to reproduce | Low/Med/High |
| User Impact | Potential harm to users or protected groups | Low/Med/High/Critical |
| Autonomy Factor | Can agents execute actions without human confirmation? | None/Partial/Full |
| Blast Radius | Single user, tenant, or cross-tenant/system-wide | Narrow/Broad/Systemic |
| Reco | Time/effort to safely restore expected behavior | Easy/Moderate/Hard |

Triage SLA (Suggested)

Critical: acknowledge immediately, mitigate within 24 hours

High: acknowledge within 4 hours, mitigate within 7 days

Medium: mitigate within 30 days

Low: backlog with risk acceptance + review date

---

🚒 AI Incident Response

Red teaming finds the holes; incident response is what you do when one is exploited in production. Agentic systems need IR patterns traditional runbooks don't cover — because a compromised agent can *act*, not just emit text.

Containment Patterns for Compromised Agents

Kill-switch — a single control that halts an agent (or agent class) immediately. Test that it actually stops in-flight tool calls, not just new prompts.

Credential rotation — revoke and rotate the agent's scoped tokens the moment compromise is suspected; assume any secret the agent could read is burned.

Memory / context quarantine — freeze and snapshot agent memory before reset, so poisoned state can be analyzed and provably purged (ties to Memory Poisoning).

Tool/MCP disablement — disable the specific tool or MCP server in the blast path while keeping the rest of the system running.

Session isolation — terminate affected sessions and prevent cross-session/context bleed.

Escalation Logic (tied to the Harm Se & Triage Model)

| Trigger | Se | Response |
|---------|----------|----------|
| Autonomous unsafe tool action (full autonomy, broad blast radius) | Critical | Kill-switch + rotate creds + page on-call immediately |
| Confirmed cross-tenant data leakage | Critical | Contain + legal/privacy notification path |
| Repeatable jailbreak family in production | High | Disable affected flow, hotfix, regression-test |
| Single-user policy , narrow blast radius | Medium | Standard ticket + scheduled fix |

Regulatory Reporting (don't skip this)

Under the EU AI Act, pro of GPAI models with systemic risk must report serious incidents to the AI Office (enforceable since 2 Aug 2026). Bake notification timelines into the runbook *before* an incident, and capture e (logs, reproductions, the report) in a form regulators and customers will accept. See Regulatory Compliance.

Post-Incident

Add the exploit to the e harness as a permanent regression test.

Run a blameless retro; feed detections back to the Purple Team loop.

Update the system's security card with the new open/closed risk.

---

🧩 Secure SDLC Integration Artifacts

To reduce "one-off" testing, integrate red-team controls into delivery workflows.

PR Security Checklist (AI Systems)

[ ] Threat model updated for new capabilities/tools

[ ] New prompts/flows added to e harness

[ ] High-risk tool actions require explicit authorization checks

[ ] Logging and privacy controls

[ ] Residual risks documented in system card

Release Readiness Criteria

No open Critical findings

All High findings ha mitigation or documented exception

Regression suite passes for required attack categories

Monitoring/detection rules deployed for new features

Operational Runbook Triggers

Sudden ASR spike (>2x baseline)

New jailbreak family with repeat success

E of cross-tenant leakage or autonomous unsafe tool use

🛡️ Defensi Patterns

Translate red-team findings into architecture decisions using a layered control model:

Reference Pipeline

Core Patterns

1. Secure Prompt Orchestration
- Separate system, de, and user instructions
- Prevent untrusted content from altering control prompts

2. Tool Permissioning and Isolation
- Grant least-pri tokens per tool and per action
- Use appro for sensiti (payments, credential resets)

3. Policy-as-Code Enforcement
- Implement deterministic checks before tool execution
- policies and test them in CI alongside prompts

4. Output Guardrails
- Add layered filters (policy, PII, compliance)
- Require citations for high-stakes domains where applicable

---

🌍 Multilingual & Cultural Safety Playbook

Test Set Design

Cover top business languages + low-resource languages in your user base

Include region-specific harmful-content categories and local legal constraints

Add culturally sensitive edge cases (slang, euphemisms, coded hate terms)

Required Test Patterns

Translation-loop bypass: blocked request translated across 2+ languages

Mixed-language prompt injection: instructions split across languages/scripts

Code-switching attacks: alternating dialect/locale per turn

Contextual harm : same request across regions with different norms

Reporting Requirements

Record language, locale, and script for every failure

Track ASR by language family to identify une co

Prioritize mitigation where user impact and language penetration are highest

---

🗂️ Data Go for Red Teaming

Data Classes in Scope

Prompts and con logs

Retrie and memory artifacts

Model outputs (including blocked/flagged outputs)

Metadata containing user identifiers or tenant references

Handling Rules (Baseline)

Minimize data collection to testing necessity

Pseudonymize/anonymize PII before long-term storage

Encrypt findings repositories and restrict access by role

Define retention windows per data class (e.g., 30/90/365 days)

Run legal/compliance review for regulated en

Go Checkpoints

Pre-engagement data handling approval

Mid-engagement privacy compliance review

Post-engagement purge and e retention sign-off

---

📊 Metrics That Matter (and Anti-Metrics)

Outcome Metrics (Use)

ASR by risk category (not only aggregate ASR)

Exploit recurrence rate after fixes

Median time-to-fix by se

Residual risk trend by quarter

Control co across high-risk abuse paths

Anti-Metrics (Avoid)

Raw number of tests executed without risk weighting

Total found as a standalone success metric

Single-point benchmark scores without trend context

“Pass rate” without confidence inter disclosure

---

🟣 Purple Team Operations

Operating Cadence

1. Red team identifies exploit chain and reproduction steps
2. Detection engineering maps telemetry and creates detections
3. Incident response drafts/updates response runbook
4. Product and platform teams ship mitigations
5. Purple-team replay detection + containment effecti

Required Outputs

Detection rule specifications linked to finding IDs

Incident runbooks for top critical/high abuse paths

Post-exercise retro: what failed, what improved, what's next

---
---

<di="center">
<a href="
<img src="assets/redteamkit-banner.svg" alt="RedTeamKit — You've read the methodology. Now run it. $249 one-time." width="100%">
</a>
</div>

---
<a id="common-implementation-pitfalls"></a>

⚠️ Common Implementation Pitfalls

| Pitfall | Why It Fails | What Good Looks Like |
|--------|---------------|----------------------|
| Keyword-only blocking | Easy to bypass /obfuscation | Semantic + policy layered controls |
| O agent tools | Enables pri escalation | Strong authz checks per tool action |
| One-time red team exercise | Misses drift and regressions | Recurring automated + manual cadence |
| Tracking only aggregate ASR | Hides high-risk hotspots | Risk-tiered metrics and trends |
| No regression suite | Reintroduces old | attack library in CI |

---

🧾 Case Study Quality Bar

Use a normalized template for all future case studies:

System context and business criticality

Attack chain with reproducible steps

Root cause and control failure points

Se and estimated remediation effort

E quality tag (E or Expert guidance)

Confidence level (High/Medium/Low)

Lessons learned and pre actions

Template a: templates/case-study-template.md

---

🪪 Model & System Cards for Security Posture

Document security posture using a structured card for every production AI system:

Intended use and prohibited use

Attack surface summary

Tested risk categories and latest date

Open risks and compensating controls

Incident escalation owners and contacts

Template a: templates/model-system-security-card.md

---

🔄 Source Hygiene & Update Go

Go Practices

Maintain a changelog for the guide (CHANGELOG.md)

Track external references with "last " timestamps

Mark major claims as E or Expert guidance

Run a quarterly review for stale links/tools/framework updates

Reference index a: resources-.md

Latest Update Watchlist (: 2026-10-01)

Use this list during quarterly maintenance to keep the guide synchronized with official sources:

1. EU AI Act — GPAI enforcement (incl. fines) and Art. 50 transparency in force since 2 August 2026. The Digital Omnibus on AI (in force 27 July 2026) mo high-risk obligations to 2 Dec 2027 and product-embedded high-risk to 2 Aug 2028. Track the GPAI Code of Practice and harmonized standards.
2. FTC in of OpenAI, Anthropic, and METR (opened late Sept 2026) o incidents and safety/assurance claims — watch for findings that affect how red-team results and third-party assessments may be described.
3. OWASP GenAI Security Project — 2026 LLM Top 10 (rebuilt with real incident data), Top 10 for Agentic Applications (ASI01–ASI10, mapped throughout this guide), the new Agent Control Standard, and the first AI Red Teaming Landscape / Solutions Directory.
4. MITRE ATLAS v5.x — 16 tactics / 80+ techniques, with agent-focused techniques such as *Publish Poisoned AI Agent Tool* and *Escape to Host*. Re-map attack trees when new ship.
5. Microsoft Taxonomy of Failure Modes in Agentic AI v2.0 (June 2026) — re-check for v2.x.
6. NIST Cyber AI Profile (IR 8596) — still a preliminary draft as of Oct 2026 (the expected summer release has not landed); workshop feedback is summarized in NIST IR 8607. Will reorganize AI cyber risk under CSF 2.0 outcomes.
7. NIST COSAiS — SP 800-53 control o for AI — single-agent and multi-agent o still in de; only the predictive-AI annotated outline has been published.
8. NIST AI RMF Profile for Trustworthy AI in Critical Infrastructure — concept note released 7 April 2026.
9. MCP & A2A security — MCP CVEs keep landing (classic web bugs dominate) and runtime-gated poisoning is now seen in the wild (Deadbugz, Aug 2026); A2A reached v1.0 under the Linux Foundation. Monitor both specs' security ad.
10. NIST SSDF SP 800-218 Rev.1 (SSDF v1.2) — re-check draft status; relevant for linking AI red-team controls to secure SDLC.

---

📎 Practitioner Appendices

Starter artifacts in templates/:

Threat modeling workshop

AI security PR checklist

Rules of engagement

report

Test case library starter

Stakeholder readout outline

Model/system security card

Case study template

📋 Regulatory Compliance

United States

#### Executi on AI (October 2023) — *historical*
The rescinded 2023 order is kept here for its widely-cited definition. It defined AI red teaming as "a structured testing effort to find flaws and in an AI system, often in a controlled en and in collaboration with de of AI. Artificial Intelligence red-teaming is most often performed by dedicated 'red teams' that adopt ad methods to identify flaws and , such as harmful or discriminatory outputs from an AI system, unforeseen or undesirable system behaviors, limitations, or potential risks associated with the misuse of the system."

What it required (no longer in effect): red teaming and reporting for dual-use foundation models, pre-deployment testing, ongoing monitoring, and incident reporting.

> Federal AI policy shifted after 2023 (the original order was rescinded and replaced by later executi). The durable US signal is now at the state level, sector regulators, and consumer-protection enforcement — track those below rather than any single executi.

#### FTC Probe of Frontier Labs and Assessors (September 2026)
The FTC opened a consumer-protection in into OpenAI, Anthropic, and METR o incidents and the safety claims made about them — the first US enforcement effort built around agents acting beyond their operators' intent. Ci Demands are expected to co records, executi, and the role of third-party assessors. Implication for red teams: your findings, scope statements, and "tested/safe" claims may become e. Write reports that state scope, co, and residual risk precisely, and ne assurance. (Washington Post · ABC News)

#### State AI Laws (2026)
With no comprehensi statute, US obligations are increasingly set by states — 45 states introduced 1,500+ AI bills in the 2025–26 sessions. The ones most relevant to security testing:

California — SB 53 (Transparency in Frontier AI Act): de of large frontier models (>10²⁶ FLOPs training compute) must publish a risk/safety framework, report critical safety incidents, and get whistleblower protections. Pairs with AB 2013 (generative-AI training-data transparency). Both effective Jan 1, 2026.

Texas — Responsible AI Go Act (TRAIGA): effective Jan 1, 2026; focuses on go use and bans manipulati uses, with lighter pri obligations.

Colorado — SB 24-205 (Colorado AI Act): the original high-risk-AI law was delayed, then enforcement was paused by a federal court, and it was superseded by SB 26-189 (signed May 2026), now effective Jan 1, 2027. Watch this one — the substance is still shifting.

Why it matters for red teams: "frontier" transparency and critical-incident-reporting duties assume you can *produce e* — documented ad testing, incident timelines, and residual-risk records. The templates in this guide map directly to those obligations.

---

European Union

#### EU AI Act (Regulation (EU) 2024/1689)
Article 15 requires operators of high-risk AI systems to demonstrate accuracy, robustness, and cybersecurity.

Implementation Timeline (as amended by the Digital Omnibus on AI):

2 February 2025: prohibited practices and AI literacy obligations entered into application

2 August 2025: go rules and GPAI obligations became applicable

2 August 2026 ✅ *in force*: Article 50 transparency duties apply, and the Commission/AI Office can now enforce GPAI obligations, including fines

2 December 2027: stand-alone high-risk AI obligations (Annex III: biometrics, critical infrastructure, education, employment, law enforcement, border management) — *moved from 2 Aug 2026 by the Omnibus*

2 August 2028: high-risk AI embedded in regulated products (e.g. medical devices, toys) — *moved from 2 Aug 2027 by the Omnibus*

> Digital Omnibus on AI (published 24 July 2026, in force 27 July 2026) deferred the high-risk timeline because harmonized standards and national authorities weren't ready — the requirements themselves are unchanged. GPAI enforcement and transparency duties were not delayed. Red teams supporting high-risk systems should use the extra time to build e, not pause testing.

##### GPAI Systemic-Risk Obligations (enforceable since 2 Aug 2026)
A general-purpose AI model is presumed to carry systemic risk when training compute exceeds 10²⁵ FLOPs; pro must notify the Commission within 2 weeks of meeting that threshold. Systemic-risk pro must then:

Conduct and document ad testing (red teaming) before placing the model on the market

Report serious incidents to the AI Office (see AI Incident Response)

Maintain cybersecurity protections for the model and its weights

Perform and document model e

The GPAI Code of Practice is the main route to demonstrate compliance ahead of harmonized standards.

##### Article → Red-Teaming Requirement → E Artifact
Map obligations to artifacts you already produce with this guide's templates:

| EU AI Act obligation | Red-teaming requirement | E artifact (template) |
|----------------------|-------------------------|------------------------------|
| Art. 15 robustness & cybersecurity | Ad testing across attack categories | report + harness ASR trends |
| GPAI systemic-risk ad testing | Documented pre-market red team with scope & results | Rules of Engagement + final report |
| Serious-incident reporting | IR runbook + notification timeline | AI Incident Response records |
| Risk management & monitoring | Continuous regression + posture tracking | Model/system security card |
| Technical documentation | Methodology, co, residual risk | Stakeholder readout + changelog |

High-Risk Systems Include: biometric identification · critical infrastructure management · educational/employment assessment · law enforcement · migration/border control · justice administration.

References: EU GPAI provider guidelines · AI Act o · Freshfields — the final Digital Omnibus on AI · Jones Walker — why 2 August 2026 still matters

---

Industry Standards

#### ISO/IEC 23894
Focuses on management of risk in AI systems, pro international standards for ensuring safety, security, and reliability.

Key Components:

Continuous testing throughout lifecycle

Red teaming methodologies

Risk management frameworks

Documentation requirements

#### ISO/IEC 42001:2023 — AI Management System (AIMS)
The first certifiable AI management-system standard (the "ISO 27001 for AI"). It requires organizations to operate a risk-based lifecycle with impact assessments, controls, and continual impro — red-team findings and remediation e are a natural fit for its Annex A controls and management review. In 2026 it is increasingly the certification enterprises and procurement teams ask for, and red-teaming platforms now map results to it alongside NIST AI RMF, OWASP, and the EU AI Act.

#### ISO/IEC 42005:2025 — AI System Impact Assessment
Provides a structured process for documenting AI system impacts (including safety/security harms). Use it to frame *what could go wrong and to whom* before scoping a red-team engagement, and to record residual risk after remediation.

---

Model Provider Requirements

#### OpenAI
"Red-team your application to ensure protection against ad input, testing the product over a wide range of inputs and user behaviors, both a representative set and those reflective of someone trying to break the model."

#### Google Gemini
"The more you red-team it, the greater your chance of spotting problems, especially those occurring rarely or only after repeated runs."

#### Anthropic
Emphasizes challenges in red teaming AI systems including:

Defining harmful outputs

Measuring rare events

E threat landscape

Resource requirements

#### Amazon Bedrock
Recommends ad testing before deployment and continuous monitoring in production.

---

📚 Resources and References

Official Frameworks

NIST AI Resources:

AI Risk Management Framework (AI RMF)

GenAI Profile (AI 600-1)

Dioptra Testbed

ARIA Program

NIST AI RMF Playbook

SP 800-218A (SSDF Community Profile for GenAI)

SP 800-218 Rev.1 Draft (SSDF v1.2)

OWASP:

GenAI Red Teaming Guide

LLM Top 10

AI Security & Privacy Guide

Top 10 for Agentic Applications 2026

MITRE:

ATLAS Framework

ATLAS Tactics

Case Studies

Cloud Security Alliance:

Agentic AI Red Teaming Guide

AI Safety Initiative

---

Academic Papers

Must-Read Papers:

1. "Lessons From Red Teaming 100 Generative AI Products" (Microsoft, 2025)
- arxiv.org/abs/2501.07238
- Real-world insights from Microsoft's red team

2. "OpenAI's Approach to External Red Teaming" (OpenAI, 2025)
- arxiv.org/abs/2503.16431
- Methodology and best practices

3. "Red Teaming AI Red Teaming" (2025)
- arxiv.org/abs/2507.05538
- Critical analysis of current practices

4. "Red-Teaming for Generative AI: Sil or Security Theater?" (2024)
- arxiv.org/abs/2401.15897
- Case study analysis

5. "A Red Teaming Roadmap" (2025)
- arxiv.org/abs/2506.05376
- Comprehensi taxonomy

---

2026 Threat-Landscape Sources

These back the 2025–2026 incidents, statistics, and framework updates added in the June 2026 refresh. /researcher-reported figures are directional, not audited.

Microsoft — Updating the taxonomy of failure modes in agentic AI (June 2026)

OWASP Top 10 for Agentic Applications 2026

EU — Guidelines for pro of general-purpose AI models

NIST — Cyber AI Profile (IR 8596 preliminary draft) · NIST IR 8607 — Cyber AI Profile workshop summary

Adversa AI — Top AI Security Incidents of 2025 · CSO Online — Top 5 real-world AI security threats of 2025

Securiti — The Anthropic exploit: era of AI agent attacks

Agentic AI red teaming reveals zero-click HITL bypass chains

Help Net Security — AI red-teaming agents change how LLMs get tested · 2026 tool landscape (Garak/PyRIT/Promptfoo)

Cisco AI Defense: Explorer Edition (agentic red teaming)

---

Tools and Platforms

Open-Source:

PyRIT - Microsoft's toolkit

Garak - LLM scanner (NVIDIA)

DeepEval - Testing framework

ART - IBM's toolkit

Giskard - AI testing platform

Gideon - Autonomous defensi assistant

Redamon - Autonomous AI red-team framework (recon → exploit → triage → auto-remediation)

AI-Infra-Guard - Full-stack AI/MCP/agent security scanner (Tencent)

Humanbound - AI-agent red-team engine, SDK, and CLI

Scenario - Simulation-based multi-turn agent red-teaming (LangWatch)

promptfoo - CI/CD-friendly LLM red teaming and evals (MIT)

BrokenHill - Automated jailbreak generator (Bishop Fox)

Counterfit - Microsoft's ML attack CLI

Darkmoon - Self-hosted autonomous AI pentesting over MCP

MiDojo - Man-in-the-middle red teaming for AI agents (asago / Red Hat)

Ziran - Graph-based tool-chain and multi-agent security testing (TaoQ AI)

Commercial:

⭐ A by Cogensec - Featured commercial platform for autonomous ad , reproducible e, and actionable remediation; frontier access by in.

Mindgard

Lakera Guard

Adversa AI

Pillar Security

Splx AI

NeuralTrust

General Analysis - Agentic + tool/MCP red teaming, CI/CD gates

Haize Labs - Large-scale automated LLM stress-testing

Verno Labs

DeepKeep AI Security Platform - Automated AI red teaming for compliance co plus Vibe AI Red Teaming for human-steered adapti

Emerging (agent-native):

Cisco AI Defense — Explorer Edition

Novee AI - Autonomous red teaming for multi-agent pipelines

---

Community and Learning

Practice Platforms:

Lakera Gandalf - Prompt injection challenges

PromptArmor - Security exercises

AI CTF - Capture the flag competitions

HackAPrompt - Prompt-hacking competitions and a large public dataset of real attacks

AI Bug Bounty Programs (scope and rewards change — read each program's current rules before testing):

Google AI Reward Program - Covers Google's AI products, including prompt-injection and data-exfiltration issues

Microsoft AI Bounty (Copilot) - AI features across Microsoft Copilot experiences

OpenAI Bug Bounty - Security issues in OpenAI's systems ()

Anthropic Bug Bounty - Security issues and safety bypasses ()

> Bug bounties are a good source of real-world attack ideas and a safe, authorized way for your team to practice. Stay inside published scope — the Disclaimer at the end of this guide applies.

Communities:

OWASP LLM Working Group - Slack channel #team-llm-redteam

AI Security Forum

AI (DEF CON)

MLSecOps community

Training:

Lakera Academy

Adversa AI courses

SANS AI security training

Academic courses on ad ML

---

Blogs and Articles

Recommended Reading:

Microsoft Security Blog - AI Red Teaming

Lakera AI Security Blog

Anthropic Safety Research

OpenAI Safety

Google AI Safety

NeuralTrust AI Security Blog

---

Books

Essential Reading:

"Ad Machine Learning" by Anthony Joseph et al.

"AI Security" by Clarence Chio & Da

"Practical AI Security" by Himanshu Sharma

"Machine Learning Security Principles" by Gary McGraw et al.

---

🤝 Contributing

We welcome contributions from the community to keep this guide comprehensive and up-to-date!

> 🌐 Beyond this repo: join the Cogensec Global Red Teaming Network to collaborate with practitioners worldwide.

How to Contribute

1. Submit Issues: Found an error or have a suggestion? Open an issue
2. Pull Requests: Add new sections, tools, or case studies
3. Share Experiences: Add your red team experiences (anonymized)
4. Update Tools: Keep tool information current
5. Add Resources: Share papers, articles, or tutorials

Contribution Guidelines

Provide sources for all claims

Include practical examples where possible

Maintain consistent formatting

Respect responsible disclosure

Avoid sharing zero-days or acti

Translations

This guide is a in multiple languages: English · Español · 中文 · Français.

English (README.md) is the source of truth. Translations are point-in-time snapshots and may lag; when they diverge, English prevails.

To add a language, copy README.md to README.<lang>.md (e.g., README.de.md), translate prose while leaving code blocks, commands, tool names, badge URLs, links, and <a id="..."> anchors unchanged, and add the new language to every language bar.

To update a translation, sync it to the latest English and update its sync note.

---

📖 Glossary

Ad Examples: Inputs crafted to fool AI systems into making incorrect predictions

Ad Training: Training technique using ad examples to impro

Attack Surface: All possible points where an AI system can be attacked

Attack Success Rate (ASR): Percentage of successful attacks attempts

Backdoor Attack: Hidden functionality triggered by specific inputs

Black Box Testing: Testing without internal knowledge of the system

Blue Team: Defensi team

Data Poisoning: Corrupting training data to compromise model

Differential Privacy: Mathematical framework for privacy protection

Emergent Behavior: Unexpected capabilities arising in AI systems

Fine-Tuning: Adapting pre-trained model to specific task

Gray Box Testing: Testing with partial knowledge of system

Guardrails: Safety mechanisms pre harmful outputs

Hallucination: AI generating false or nonsensical information

Jailbreaking: Bypassing AI safety restrictions

Membership Inference: Determining if data was in training set

Model Extraction: Stealing AI model through queries

Model In: Reconstructing training data from model

Multimodal: AI processing multiple types of input (text, image, audio)

Prompt Injection: Manipulating AI through crafted prompts

Purple Team: Collaborative red and blue team approach

RAG (Retrie Generation): AI augmented with external knowledge

Red Team: Offensi team simulating attacks

RLHF (Reinforcement Learning from Human Feedback): Training technique using human preferences

Shadow Model: Surrogate model mimicking target system

Supply Chain Attack: Compromising AI through dependencies

White Box Testing: Testing with full internal knowledge

Zero-Day: Pre unknown

---

📄 License

This guide is released under the MIT License. Feel free to use, modify, and distribute with attribution.

---

🙏 Acknowledgments

This guide draws from research and best practices established by:

Microsoft AI Red Team - For pioneering enterprise-scale AI red teaming

OpenAI - For transparency in red team methodologies

OWASP Foundation - For the GenAI Red Teaming Guide

NIST - For comprehensive AI Risk Management Framework

MITRE Corporation - For the ATLAS knowledge base

Cloud Security Alliance - For Agentic AI guidance

Anthropic - For ethical AI safety research

Academic Researchers - For ad ad ML science

Contributors

— promptfoo, LLM red teaming and e (#1)

— NeuralTrust, AI red teaming services and generati firewall (#2, #3)

— Pallma AI, later rebranded Verno Labs (#7, #14)

— Redamon, autonomous AI red-team framework

— DeepKeep AI Security Platform (#21)

@MBK-fr — Darkmoon, self-hosted autonomous AI penetration testing (#23)

— Ziran, graph-based tool-chain and multi-agent security testing (#22, landed via #27)

---

📞 Contact

For Questions or Feedback:

Open an issue on GitHub

Connect with the AI security community

For Security :

Follow responsible disclosure practices

Contact security teams directly

Use coordinated disclosure timelines

---

<di="center">

---

<di="center">

🛡️ You've read the methodology. Now run it.

RedTeamKit is the implementation layer for this guide — 7 production npm packages,
scoped assessment templates, prompt injection payloads, and reporting scaffolds
used in real AI security engagements.

Ship your first assessment this week, not this quarter.

*$249 one-time · Lifetime updates · Built by the author of this guide*

</div>

---

</div>

> ⚠️ Authorized use only. Use RedTeamKit exclusively on systems you own or are explicitly authorized to test.

---

<di="center">
<a href="
<img src="assets/redteamkit-banner.svg" alt="RedTeamKit — You've read the methodology. Now run it. $249 one-time." width="100%">
</a>
</div>

---
---

⚠️ Disclaimer

This guide is for educational and security research purposes. All testing should be conducted:

With proper authorization

On systems you own or ha to test

In compliance with applicable laws and regulations

Following ethical guidelines

Unauthorized testing of AI systems may be illegal and unethical. Always obtain explicit permission before conducting red team exercises on systems you do not own or control.

---

<di="center">

🎯 Remember: Responsible red teaming makes AI safer for e 🎯

Last Updated: October 2026

Star this repository to stay updated with the latest AI red teaming practices!

Star History

![Star History Chart]()
</div>

项目信息

开发语言：未知

Star 数：902

技术要点

该项目属于红蓝攻防方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕A comprehensive guide to adversarial testing and security evaluation of AI systems, helping organizations identify vulnerabilities before attackers exploit them.展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。

登录后才能参与讨论

下载链接

会员才能下载

登录后开通会员

同分类推荐

暂无同类资源

资源标题	AI-Red-Teaming-Guide - A comprehensive guide to adversarial testing and s
所属分类	红蓝攻防
发布作者	admin
发布时间	2026-10-06 00:37:03
下载权限	免费下载
附件数量	2 个下载链接
浏览次数	497
讨论数量	0
主题状态	正常讨论中