dspy-redteam - Red-Teaming Language Models with DSPy

项目说明

Red-Teaming Language Models with DSPy

We use the the power of DSPy, a framework for structuring and optimizing language model programs, to red-team language models.

To our knowledge, this is the first attempt at using any auto-prompting *framework* to perform the red-teaming task. This is also probably the deepest architecture in public optimized with DSPy to date.

We accomplish this using a *deep* language program with several layers of alternating Attack and Refine modules in the following optimization loop:

<figure style="text-align: center;">
<img src="/images/DSPy-Redteam.png" alt="O of DSPy for red-teaming" style="margin: 0 auto; margin-bottom: 20px; display: block;">
<figcaption><i>Figure 1: O of DSPy for red-teaming. The DSPy MIPRO optimizer, guided by a LLM as a judge, compiles our language program into an effecti against .</i></figcaption>
</figure>

The following Table demonstrates the effecti of the chosen architecture, as well as the benefit of DSPy compilation:

<di="center">

| Architecture | ASR |
|:------------:|:----------:|
| None (Raw Input) | 10% |
| Architecture (5 Layer) | 26% |
| Architecture (5 Layer) + Optimization | 44% |

<em>Table 1: ASR with raw harmful inputs, un-optimized architecture, and architecture post DSPy compilation.</em>
</div>

With *no specific prompt engineering*, we are able to achieve an Attack Success Rate of 44%, 4x over the baseline. This is by no means the SOTA, but considering how we essentially spent no effort designing the architecture and prompts, and considering how we just used an off-the-shelf optimizer with almost no hyperparameter tuning (except to fit compute constraints), we think it is pretty exciting that we can achieve this result!

Full exposition on the Haize Labs blog.

项目信息

开发语言：Python

Star 数：275

技术要点

该项目属于红蓝攻防方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Red-Teaming Language Models with DSPy展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。