AIRTBench-Code - 代码库： AIRTBench ： Measuring Autonomo

项目说明

AIRTBench: Autonomous AI Red Teaming Agent Code

<di="center">

<img
src="
alt="Logo"
align="center"
width="144px"
height="144px"
/>

</div>

<!-- BEGIN_AUTO_BADGES -->
<di="center">

![Pre-Commit]()
![Renovate]()
![License]()
![GitHub release (latest by date)]()

![arXiv]()
![HuggingFace]()
![Dreadnode]()
![Agent Harness]()

![GitHub stars]()
![PRs Welcome]()

</div>
<!-- END_AUTO_BADGES -->

---

This repository contains the implementation of the AIRTBench autonomous AI red teaming agent, complementing our research paper AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models and accompanying blog post, "Do LLM Agents Have AI Red Team Capabilities? We Built a Benchmark to Find Out".

The AIRTBench agent is designed to e the autonomous red teaming capabilities of large language models (LLMs) through AI/ML Capture The Flag (CTF) challenges. Our agent systematically exploits LLM-based targets by solving challenges on the Dreadnode Strikes platform, pro a standardized benchmark for measuring ad AI capabilities.

AIRTBench: Autonomous AI Red Teaming Agent Code

- Agent Harness Construction
- Setup
- Documentation
- Run the E
- Basic Usage
- Challenge Filtering
- Resources
- Dataset
- Citation
- Model requests
- 🤝 Contributing
- 🔐 Security
- ⭐ Star History

Agent Harness Construction

The AIRTBench harness follows a modular architecture designed for extensibility and e:

<di="center">
<img src="assets/airtbench_architecture_diagram_dark.png" alt="AIRTBench Architecture" width="100%">
<br>
<em>Figure: AIRTBench harness construction architecture showing the interaction between agent components, challenge interface, and e framework.</em>
</div>

Setup

You can setup the en with uv:

Documentation

Technical documentation for the AIRTBench agent is a in the Dreadnode Strikes documentation.

Run the E

<mark>In order to run the code, you will need access to the Dreadnode strikes platform, see the docs or submit for the Strikes waitlist here</mark>.

This rigging-based agent works to solve a of AI ML CTF challenges from the dreadnode Crucible platform and gi to execute python commands on a network-local container with custom Dockerfile.

Basic Usage

Organization and Workspace Parameters

If you are part of multiple organizations, you must specify which organization to use:

Optionally, you can also specify a workspace within the organization:

Example with organization:

Challenge Filtering

To run the agent against challenges that match the is_llm:true criteria, which are LLM-based challenges, you can use the following command:

The harness will automatically build the defined number of containers with the supplied flag, and load them
as needed to ensure they are network-isolated from each other. The process is generally:

1. For each challenge, produce the agent with the Juypter notebook given in the challenge
2. Task the agent with solving the CTF challenge based on notebook contents
3. Bring up the associated en
4. Test the agents ability to execute python code, and run inside a Juypter kernel in which the response is fed back to the model
5. If the CTF challenge is solved and flag is observed, the agent must submit the flag
6. Otherwise run until an error, give up, or max-steps is reached

Check out the challenge manifest to see current challenges in scope.

Resources

📄 Paper on arXiv

📝 Blog post

Dataset

Download the dataset directly from 🤗Hugging Face

Instructions for loading the dataset can be found in the dataset directory also.

Citation

If you find our work helpful, please use the following citations.

Model requests

If you know of a model that may be interesting to analyze, but do not have the resources to run it yourself, feel free to open a feature request via a GitHub issue.

🤝 Contributing

Forks and contributions are welcome! Please see our Contributing Guide.

🔐 Security

See our Security Policy for reporting .

⭐ Star History

![GitHub stars]()

By watching the repo, you can also be notified of any upcoming releases.

![Star history graph]()

项目信息

开发语言：Jupyter Notebook

Star 数：112

技术要点

该项目属于CTF思路方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Code Repository for: AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。