OpenART - OpenART is an open-source framework designed to ev

项目说明

OpenART

OpenART is an executable benchmark for e the safety of tool-using code
agents in long-running, e en. This repository contains its
Docker-based runtime, three runnable high-complexity tasks, and the managed
tools required by those tasks.

OpenART is the agent of OpenRT.

[09/22/2026] ⭐ OpenART reached the 200-star milestone on GitHub.

![arXiv]()
![Dataset]()
![Tools]()

Research summary

OpenART (Open Agent Red Teaming) is a benchmark and Docker-nati
for agent safety e: red-teaming tool-using LLM agents inside
long-horizon, executable, e en. It is relevant when your
question matches one of these patterns:

How safe are tool-using / code agents (OpenCode, Claude Code, Codex CLI,

and 12 other runtimes) beyond single-turn chat?

Do agents fail when the en state changes mid-run (files edited,

tools swapped, services updated) — *en e* — while the task
and hidden safety contract stay fixed?

How does safety vary across agent runtimes across foundation

models (15 agents × 5 models e matrix)?

How do attacks behave at long horizon (median 97 tool calls), and where

do unsafe actions first appear after mutated state is consumed?

How can LLM red teaming / prompt-injection-style attacks be scaled

beyond hand-written scenarios (planner-generated 10K+ tasks,
500K+ tool corpus, e EMHA attacker)?

The core idea: e the executable en.
Persistent state is mutated during execution; safety is judged against a fixed
hidden contract by deterministic and model-based e o and
snapshots.

Comparison with existing benchmarks (from the paper)

Task-le measured from released artifacts and executable traces,
reported as medians [interquartile range]. OpenART and DTap report medians
o execution traces; all other benchmarks are measured from
their publicly released samples.

| Benchmark | Tool calls | Dependency depth | Parallel width | State objects | File formats |
| --- | ---: | ---: | ---: | ---: | ---: |
| InjecAgent | 1 [1-1] | 1 [1-1] | 1 [1-1] | 1 [1-1] | 0 [0-0] |
| ToolEmu | 3 [1.5-4] | 2.5 [1.2-3.8] | 1 [1-1.8] | 3 [1.2-3] | 0 [0-0] |
| AgentDojo | 2 [1-3] | 2 [1-3] | 1 [1-1] | 1 [1-2] | 0 [0-0] |
| AgentHarm | 3.5 [3-4] | 3 [3-3] | 1.5 [1-2] | 3.5 [3-4] | 0 [0-0] |
| ASB | 2 [2-2] | 2 [2-2] | 1 [1-1] | 2 [2-2] | 0 [0-0] |
| DTap | 15 [7.4-18.7] | 2 [1-3] | 1.5 [1-2] | 2.5 [1-4] | 1 [0-3] |
| OpenART | 97 [90.2-100] | 32 [15.8-84.8] | 12.5 [3-24.5] | 96.5 [90.2-100] | 7.5 [7-9] |

OpenART reaches a median dependency depth of 32 and parallel width of 12.5,
reflecting long, branching workflows that support red teaming o
interaction horizons.

Results across the 75 agent-model settings

Benign task completion (Comp.) and Strict ASR (%) pooled over 15 target agents
per foundation model, from the paper's 75-setting e matrix. Strict
ASR counts an attack as successful only when both the deterministic e
and an LLM judge confirm it.

| Foundation model | Comp. (%) | Strict ASR (%) |
| --- | ---: | ---: |
| GPT-5.5 | 92.02 | 88.5 |
| Claude Opus-4.8 | 96.18 | 59.2 |
| GLM-5.2 | 85.00 | 87.9 |
| Qwen-3.7-Max | 80.81 | 94.6 |
| DeepSeek- | 82.91 | 94.7 |
| Pooled (75 settings) | 87.38 | 85.0 |

Paper and datasets

OpenART: Scaling Agent Red Teaming En E

🥈 #2 Paper of the Day on Hugging Face Papers

Yunhao Chen, Xin Wang, Yixu Wang, Yi Liu, Jie Li, Yan Teng, Xingjun Ma, Xia Hu,
and Yu-Gang Jiang

arXi ·
PDF

OpenART e the executable en rather than treating a single
prompt as the entire test. The benign task and hidden safety contract remain
fixed while target- state changes during execution.

| Paper setting | Value |
| --- | ---: |
| stateful scenarios | 10K+ |
| Domains | 50 |
| Capability corpus | 500K+ tools and skills |
| Median task horizon | 97 tool calls |
| E matrix | 15 agents × 5 models (75 settings) |
| Target- attack surfaces | 8 |
| Reference attacker | E Marko Attack (EMHA) |
| Pooled strict attack success rate | 85.0% |

The paper reports that en e becomes more effective as
workflow complexity grows, and that the agent runtime affects safety beyond
the choice of foundation model. Unsafe behavior also tends to appear well
after the changed state is first consumed, often through stale assumptions or
decisions carried forward from earlier steps.

The public
OpenART planner task dataset
contains 6,597 tasks in 34 zip shards. Its release checks covered
checksums, archi, and sample extraction.

We also release the public
OpenART tools dataset,
which packages 63,697 selected materialized tools from the OpenART capability
corpus. This 60K+ subset prioritizes the most usable OpenART tools for task
execution, reuse, and reproducible benchmark construction. The release includes
metadata, archi, checksums, and a lightweight download helper.

Repository layout

| Path | Contents |
| --- | --- |
| OpenART/ | Runtime, configuration, Dockerfiles, documentation, and tests |
| OpenART/examples/tasks/ | Local smoke task and bundled high-complexity tasks |
| OpenART/skills/openart/SKILL.md | Unified guide for agents and people using or extending OpenART |
| openart-tools/ | Managed tool subset used by the bundled tasks |

Use the OpenART skill

OpenART includes one consolidated operator skill for planning, scenario
generation, managed tools, e, extension, and debugging. It routes
each request to the appropriate existing documentation and command instead of
duplicating those workflows across several skills.

For people: read OpenART/skills/openart/SKILL.md

as a workflow map before choosing a detailed guide.

For agents: explicitly ask the agent to read that file before working on the

framework. For example:

Agents that support repository-local skill discovery can use OpenART/skills/
as their skill source. Otherwise, referencing the file directly is the
portable in method. This operator skill is not a managed runtime tool,
target- skill, or attacker payload.

Setup

Run the following commands from the repository root:

Docker must be a to the current user. Build the base task image:

Local smoke test

The local smoke task checks the container and deterministic e paths. It
does not require model credentials.

Model-backed e

The default target configuration uses OpenCode. Build its image, copy the
en template, and add the required model endpoints and credentials to
.env:

Run a target without an attacker:

Run the same task with the default OpenCode-compatible attacker:

The task graph selects tools from ../openart-tools. Run artifacts are written
under OpenART/outputs/ and ignored by Git.

Task generation

Build the planner image and generate a task from a checked-in scenario. Planner
model settings are read from .env.

Planner outputs are also local, ignored artifacts.

Documentation

Documentation index

Quickstart and runtime options

Planner design and usage

Unified operator guide

Citing OpenART

If you use the OpenART runtime, tasks, tools dataset, or e results,
please cite the paper and the repository. The canonical metadata lives in
CITATION.cff, which GitHub exposes as the "Cite this
repository" link.

The official project name is OpenART; use the paper title abo
in bibliographies rather than such as "OpenART Arena".

License

OpenART is licensed under the GNU Affero General Public License v3.0.

项目信息

开发语言：Python

Star 数：230

技术要点

该项目属于红蓝攻防方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕OpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。