GPTFuzz - Official repo for GPTFUZZER : Red Teaming Large La

项目说明

GPTFUZZER : Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts

<img src="./sources/icon.png" width=500>

![License: MIT]()

This is the official repository for "GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts" by Jiahao Yu, Xingwei Lin, Zheng Yu, Xinyu Xing.

Table of Contents

Updates

Installation

Datasets

Models

Running

Release

FQA

Updates

(2024/8/13) This paper was accepted by Usenix Security. See the proceeding paper

(2023/10/25) 🏆 🏆 We won the Frontier Breakthrough Award and Outstanding Presentation Award at Geekcon 2023!

(2023/10/22) Our work will be presented in the Geekcon 2023 conference! We will give a talk about our work and give a li demo with our tool. We will also include the attack results for Chinese LLMs which have not be included in our paper. We are looking forward to meeting you there!

(2023/10/21) We ha our codebase to make it more readable and easier to use. We will continue to update the codebase and add more features and other implementations. We are looking forward to build a general black-box fuzzing framework for large language models. Stay tuned!

(2023/9/19) Our paper is on arXi it out here!

Installation

Please refer to install.ipynb

Datasets

The datasets for the harmful question and human-written templates are a in datasets/questions/question_list.csv and datasets/prompts/GPTFuzzer.csv. The questions are sampled from two public datasets: llm-jailbreak-study and hh-rlhf, and the templates are collected from llm-jailbreak-study.

For the responses we got by querying , ChatGPT and Llama-2-7B-chat, we store them in datasets/responses and the labeled responses are in datasets/responses_labeled. You could also use generate_responses.py to generate responses for different models or different questions (see the scripts under scripts folder for examples).

We are still working on the e on other question dataset and jailbreak dataset. We will update the codebase and the datasets after we have some results.

Models

Our judgment model is a finetuned RoBERTa-large model and the training code is in ./example/finetune_roberta.py, and the training/e data is stored in datasets/responses_labeled. The model we used is hosted on Hugging Face. When running fuzzing experiments, the model will be automatically downloaded and cached for the first time. If you would like to download the model manually, you can run the following code:

During our experiments, we found that our trained model can also be transferred to other questions. However, we also found that it does not work well on some questions and other languages. We will add more predictor model soon.

Running

We provide a python example to show the minimal code to run the fuzzing experiments. This example uses ChatGPT as mutate model to attack Llama-2-7B-chat with official system prompt(we did the monkey patch for Fastchat template since Fastchat deleted the official system prompt in recent update), and you should be able to get the identical results in example folder (we set the random seed for reproducibility and temperature=0).

You can also refer to the notebook for more details and explanations.

Release

Due to ethical concern, we decided not to release the ad templates we found during our experiments openly. However, we are happy to share them with researchers who are interested in this topic. Please contact us via email if you would like to get access to the templates we found during the experiments. Also, you can use the code in this repository to generate your own ad templates.

FQA

1. I found some labels in your labeled responses are wrong.
- We are sorry about that. As our paper claimed, determining whether it is a jailbroken response is not a trivial task and some responses are hard to label. Also, due to the stress of labeling a large amount of potential toxic responses, we might have made some mistakes. If you find any wrong labels, please let us know and we will fix them as soon as possible.
2. The fuzz is slow, especially when I am using multiple questions for the local model.
- We found that use batched inference can significantly speed up the fuzzing process. However, the results might be slightly different from the original results because of the padding tokens (see here). We suggest using vllm inference for hyper performance
3. How could I implement my own mutator/seed selector?
- You can implement your own mutator/seed selector by inheriting the class. You can refer to mutator.py and selection.py for examples.
Also, as we claimed, we would like to work on a general black-box fuzzing framework for large language models. If you have some ideas or suggestions or you find other papers that are related to this topic, please let us know or leave the comment in the issue. We are happy to implement them and make this framework more powerful.

Citation

If you find this useful in your research, please consider citing:

项目信息

开发语言：Python

Star 数：611

技术要点

该项目属于红蓝攻防方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Official repo for GPTFUZZER : Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。