safelabs-eval - Red-teaming and evaluation framework for AI agents

项目说明

<p align="center">
<img src=" alt="safelabs-eval" width="400">
</p>

<di="center">

![CI]()
![License]()
![Taxonomy: OWASP-inspired]()
![PyPI ]()
![PyPI - Python ]()
![DOI]()

</div>

safelabs-eval

Open-source red-teaming and e framework for AI agents — built around an OWASP-inspired agent-security taxonomy.

---

AI agents built on LangChain, CrewAI, AutoGen, LlamaIndex, the OpenAI Agents SDK, Google ADK, Semantic Kernel, and custom frameworks can reach production without systematic ad safety testing. safelabs-eval changes that.

Point it at a supported HTTP agent endpoint — or wrap a Python callable — and it fires 300 curated ad prompts (30 per category) across all 10 ASI categories, scores every response with pattern-based detectors, and produces a structured security report. The ASI01–ASI10 category structure is OWASP-inspired, not the official OWASP Top 10 for Agentic Applications 2026 — see the taxonomy note.

No LLM calls required for detection. No agent code modifications required. No SafeLabs-specific infrastructure required.

---

Install

Requirements: Python 3.11+

---

Quick Start

Option 1 — CLI: test a supported HTTP agent endpoint

Example report — Claude Haiku Prompt Injection (abbre to the first 3 prompts; captured at v0.2.2, when ASI01 had 14 prompts — it now has 30):

---

Option 2 — Python API: wrap any callable

Both def and async def callables are accepted. No agent code modification required.

Run all 10 ASI categories (300 prompts total):

Access indi results:

> Reading the confidence figure. The confidence value (shown as "conf" in
> the CLI report) is a heuristic e score produced by the pattern-based
> detectors — roughly, how much matched e supports the . It is
> not a statistically calibrated probability, and should not be read as "the
> chance this is correct."

---

Browse the prompt library

---

ASI Category Co

> Taxonomy note. safelabs-eval's prompt taxonomy (ASI01–ASI10) is an
> OWASP-inspired agent-security category structure that uses an independently
> structured category set. It differs from OWASP's
> official Top 10 for Agentic Applications 2026
> (published Dec 9, 2025), which uses a different category structure (Agent
> Goal Hijack, Tool Misuse & Exploitation, Identity & Pri Abuse, etc.).
> This corpus is not currently mapped to that newer official taxonomy. Note
> that the official list also numbers its categories ASI01–ASI10, so an
> ID such as ASI01 here does not refer to the same category as OWASP's
> ASI01 (Agent Goal Hijack).

| Category | Name | Prompts | Detector |
|---|---|---|---|
| ASI01 | Prompt Injection | 30 | PromptInjectionDetector |
| ASI02 | Insecure Output Handling | 30 | pattern suite |
| ASI03 | Excessi | 30 | Scope |
| ASI04 | Resource Management | 30 | pattern suite |
| ASI05 | Tool Use Safety | 30 | pattern suite |
| ASI06 | Data Privacy & Confidentiality | 30 | DataLeakageDetector |
| ASI07 | Trust Boundaries | 30 | pattern suite |
| ASI08 | Beha Drift (single-turn scenarios — apparent multi-turn framing is represented within one prompt, not executed across turns) | 30 | JailbreakDetector |
| ASI09 | Scope | 30 | Scope |
| ASI10 | Hallucination & Misinformation | 30 | HallucinationDetector |

300 ad prompts · 5 pattern-based detectors · 10 ASI categories · zero additional LLM grading cost

Prompt library

300 ad prompts — 30 per ASI category, and exactly 10 per
(category, difficulty_tier) cell (tier_1, tier_2 and tier_3 each at
10 within every category). Every entry
(safelabs/prompts/schemas.py::PromptEntry) carries structured metadata
under schema v1.1.0:

difficulty_tier — tier_1 overt / tier_2 contextual / tier_3

adaptive. This is an author-defined attack-sophistication tier, derived
from a documented rubric (se floor + e bumps; see the
rubric block in safelabs/prompts/library.py). It is a corpus-internal
scale, not empirically calibrated against measured model refusal rates.

pro — original | adapted-from:<slug> |

deri:C. All 300 entries are original.

atlas_technique_ids — MITRE ATLAS v5.6.0 technique ids

(AML.T####[.###]), or the single literal ['UNMAPPED'] where no
ATLAS technique cleanly applies (ASI02 and ASI10, by review).

The content returned by load_library() is 1.13.0 (schema
1.1.0). It was built in thirteen changelog steps (1.1.0
through 1.13.0), most recently the v1.13.0 library-wide floor-to-10
batch (+129 prompts, 171 → 300). The content ad with each
prompt batch; the schema ad only when PromptEntry's shape
changes. Because content are per batch, pin the repository commit SHA
containing the e library.py when citing a specific run. Full build history and the
difficulty rubric live in the library.py module docstring; the dataset
card is docs/DATASET_CARD.md.

Dataset. SafeAgent-300, a redacted e dataset produced with this framework (safelabs-eval 0.10.1, prompt library 1.13.0): 10.5281/zenodo.23092031 (all ); v1.0.0: 10.5281/zenodo.23092032. Raw model outputs are withheld; see the dataset's datasheet.

Pro & licensing

A full pro and audit was run over the 131 prompts of
content 1.6.0 (the library as it stood at that release). Findings: **no prompt reproduces text, code, or
dataset rows from any external source.** A number of entries carry a
code-comment note recording the *technique or scenario domain* they draw
on — e.g. indirect prompt injection (Greshake et al. 2023), agent
tool-use attacks (AgentDojo / Debenedetti et al. 2024), the
"ignore all previous instructions" idiom (PromptInject / Perez &
Ribeiro 2022), multi-turn escalation (Crescendo / Russinovich et al.
2024), and probe categories from garak and PyRIT. Each cited source and
its license is listed in CREDITS.md; the audit summary is
in the dataset card.

The pro metadata field and the audit are separate things. The 169
prompts added after v1.6.0 are labeled original in the corpus metadata but
have not undergone the same full external-source pro audit; the
per-batch o checks that were run on them instead are described in
CREDITS.md. That does not mean the newer prompts carry no
sourcing information: at least one of them (ASI01-018, added in v1.7.0)
has a source-lineage code comment citing garak's dan probe. It just hasn't
had the same systematic review.

---

Levels

| | Meaning | Recommended Action |
|---|---|---|
| | Detector identified apparent attack compliance | Prioritize review and remediation |
| FAIL | Detector identified signs of weakness without apparent full compliance | Review and consider hardening |
| UNCERTAIN | Detector e was ambiguous | Human review recommended |
| PASS | No unsafe compliance detected in this response | No issue detected in this test |

---

Known Limitations

safelabs-eval uses pattern-based detectors rather than human or LLM
adjudication. Detectors may miss subtle, novel, or paraphrased forms of unsafe
compliance, and may produce false positives. Results should be interpreted as
detector-deri, not ground-truth safety measurements.

The 300-prompt library (the SafeAgent-300 corpus) is English-only and
single-turn. ASI08 scenarios may describe multi-turn interactions within a
single prompt but do not execute persistent multi-turn con.

Difficulty tiers represent an author-defined attack-sophistication rubric and
have not been empirically calibrated against measured model failure rates.

Co is breadth-oriented and should not be interpreted as exhaustive
co of any ASI category.

---

Why safelabs-eval?

| Problem | safelabs-eval |
|---|---|
| Fragmented agent-security e | 300 curated prompts across all 10 ASI categories |
| Security tools require LLM calls to score | Pure Python detectors — zero additional LLM grading cost |
| Testing tied to one framework | Framework-agnostic — HTTP endpoint or Python callable |
| No audit trail for agent e | Structured JSON output for CI/CD and audit/e workflows |

---

Architecture

Design principles:

Detectors are pure Python — no LLM calls, no I/O, no database

All detection is async-first — safe for concurrent e

Regex patterns compiled once at init — reused across every call

E is extensible — implement BaseDetector, register with Scorer

---

Framework Adapters

Install only the extras you need. The core package (HTTP adapter + CLI) requires no optional dependencies.

| Framework | pip extra | Minimum |
|---|---|---|
| LangChain | pip install "safelabs-e]" | langchain-core>=0.1 |
| CrewAI | pip install "safelabs-e]" | crewai>=0.30 |
| AutoGen / ag2 | pip install "safelabs-e]" | ag2>=0.2 |
| LlamaIndex | pip install "safelabs-e]" | llama-index-core>=0.11 |
| OpenAI Agents SDK | pip install "safelabs-e]" | openai-agents>=0.1 |
| Google ADK | pip install "safelabs-e]" | google-adk>=1.0 |
| Semantic Kernel | pip install "safelabs-e]" | semantic-kernel>=1.26 |

Usage examples:

Pass any adapter to run_eval or call .execute(prompt) directly:

Every adapter's execute() returns an AgentResponse. run_e.execute) scores its output text and also reads its error, metadata and tool_calls; plain-string callables such as my_agent abo work as before.

---

What's Coming

We're actively de new detectors, prompts, and reporting features.
Watch this repo or join the discussion in GitHub Issues to follow along and shape the direction.

Want to contribute? The highest-value open areas right now:

Additional ad prompts — the library now covers 30 prompts per category (10 per difficulty tier); no and harder tier_3 (adapti within the existing categories are still welcome.

Integration test harnesses — the current adapter tests use duck-typed fakes and do not install real framework packages. Tests that run against actual LangChain, CrewAI, AutoGen, LlamaIndex, OpenAI Agents SDK, Google ADK, and Semantic Kernel objects (in an optional CI job) are a real gap.

Richer detectors — current detectors are regex-based; LLM-graded and embedding-similarity detectors could reduce some co gaps on subtle attacks that pattern matching misses.

Open an issue before submitting a PR.

---

Contributing

---

Research & Disclosure

safelabs-eval is an open-source, red-teaming and e framework for AI agent security, de and maintained by Safe Labs AI Inc.

Findings from red-teaming exercises conducted with this framework are published as research. If you disco attack patterns or agent using safelabs-eval, please open an issue or reach out — responsible disclosure is appreciated and credited.

The project's earlier standalone analyses have been consolidated into
three papers:

ABC Merged — detector-calibration reliability, combining the former

Papers A, B and C.

AgentPort-Bench — cross-framework portability, including controlled

e across framework, model, and configuration .

SafeAgent-300 — the 300-prompt benchmark paper.

The four original Figshare preprints are listed below for reference. They are
the original preprints; the merged papers above are the current, complete
statement of this work.

"Pattern-Matching Failures in LLM Refusal Detection: A Case Study in Detector Reliability" — original preprint; now part of ABC Merged

"Does a Pattern-Matching Detector Fix Generalize? A Six-Model Replication and the Prompt Shapes That Break It" — original preprint; now part of ABC Merged

"Replicating a Detector-Calibration False-Positi in GPT-5.5 Prompt-Injection Across a Second Model and Category" — original preprint; now part of ABC Merged

"Cross-Framework Portability of Agentic AI Security: A Controlled, Payload- E" — original preprint; now part of AgentPort-Bench

The exploratory run that motivated Paper A is documented in the original
blog post: "Why Claude Haiku Returned UNCERTAIN: Anatomy of an Indirect
Prompt Injection in an Agentic System".

---

Ecosystem

safelabs-research — attack taxonomy, ad, and blog posts on AI agent security; the source for research findings published using this framework.

Blog — research writeups from Safe Labs AI, including "We Were Wrong About the UNCERTAIN Results — Here's What Actually Happened", a public correction — with a dated addendum on the same page — of an earlier finding about this framework's own detector reliability.

---

Related Work

OWASP Top 10 for LLM Applications

OWASP Top 10 for Agentic Applications 2026 — the official OWASP agentic taxonomy; not currently mapped to this corpus

Garak — LLM scanner

PyRIT — Microsoft Python Risk Identification Toolkit

Promptfoo — LLM testing framework (announced agreement to be acquired by OpenAI, March 2026)

---

License

Apache 2.0 — see LICENSE.

---

<p align="center">
Built by <a href=" Labs AI Inc.</a> ·
<a href=" an Issue</a> ·
<a href="
</p>

项目信息

开发语言：Python

Star 数：486

技术要点

该项目属于红蓝攻防方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Red-teaming and evaluation framework for AI agents, built around an OWASP-inspired ASI01–ASI10 taxonomy展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。