PurpleLlama - Set of tools to assess and improve LLM security.

项目说明

<p align="center">
<img src=" width="400"/>
</p>

<p align="center">
🤗 <a href=" Models on Hugging Face</a>&nbsp | <a href=" Blog</a>&nbsp | <a href=" | <a href=" E</a>&nbsp&nbsp | <a href=" Guard Paper</a>&nbsp
<br>

---

Purple Llama

Purple Llama is an umbrella project that over time will bring together tools
and evals to help the community build responsibly with open generative AI
models. The initial release will include tools and evals for Cyber Security and
Input/Output safeguards but we plan to contribute more in the near future.

Why purple?

Borrowing a concept from the
cybersecurity world, we believe that to truly mitigate the challenges which
generative AI presents, we need to take both attack (red team) and defensive
(blue team) postures. Purple teaming, composed of both red and blue team
responsibilities, is a collaborati to e and mitigating
potential risks and the same ethos applies to generative AI and hence our
in in Purple Llama will be comprehensive.

License

Components within the Purple Llama project will be licensed permissively enabling both research and commercial usage.
We believe this is a major step towards enabling community collaboration and standardizing the de and usage of trust and safety tools for generative AI de.
More concretely evals and benchmarks are licensed under the MIT license while any models use the corresponding Llama Community license. See the table below:

| Component Type | Components | License |
| :----------------- | :----------------------------------: | :--------------------------------------------------------------------------------------------: |
| Evals/Benchmarks | Cyber Security Eval (others to come) | MIT |
| Safeguard | Llama Guard | Llama 2 Community License |
| Safeguard | Llama Guard 2 | Llama 3 Community License |
| Safeguard | Llama Guard 3-8B | Llama 3.2 Community License |
| Safeguard | Llama Guard 3-1B | Llama 3.2 Community License |
| Safeguard | Llama Guard 3-11B- | Llama 3.2 Community License |
| Safeguard | Prompt Guard | Llama 3.2 Community License |
| Safeguard | Code Shield | MIT |

System-Le

As we outlined in Llama 3’s
Responsible Use Guide, we
recommend that all inputs and outputs to the LLM be checked and filtered in
accordance with content guidelines appropriate to the application.

Llama Guard

Llama Guard 3 consists of a series of high-performance input and output moderation models designed to support de to detect common types of content.

They were built by fine-tuning Meta-Llama 3.1 and 3.2 models and optimized to support the detection of the MLCommons standard hazards taxonomy, catering to a range of de use cases.
They support the release of Llama 3.2 capabilities, including 7 new languages, a 128k context window, and image reasoning. Llama Guard 3 models were also optimized to detect helpful cyberattack responses and prevent malicious 代码 output by LLMs to be executed in hosting en for Llama systems using 代码 interpreters.

Prompt Guard

Categories of prompt attacks include prompt injection and jailbreaking:

* Prompt Injections are inputs that exploit the inclusion of untrusted data from third parties into the context window of a model to get it to execute unintended instructions.
* Jailbreaks are malicious instructions designed to o the safety and security features built into a model.

Code Shield

代码 Shield adds support for inference-time filtering of insecure 代码 produced by LLMs. 代码 Shield offers mitigation of insecure 代码 suggestions risk, 代码 interpreter abuse pre, and secure command execution. CodeShield Example Notebook.

Evals & Benchmarks

Cybersecurity

#### CyberSec Eval v1
CyberSec Eval v1 was what we believe was the first industry-wide set of cybersecurity safety e for LLMs. These benchmarks are based on industry guidance and standards (e.g., CWE and MITRE ATT&CK) and built in collaboration with our security subject matter experts. We aim to provide tools that will help address some risks outlined in the White House commitments on de responsible AI, including:
* Metrics for quantifying LLM cybersecurity risks.
* Tools to e the frequency of insecure 代码 suggestions.
* Tools to e LLMs to make it harder to generate malicious 代码 or aid in carrying out cyberattacks.

We belie tools will reduce the frequency of LLMs suggesting insecure AI-generated 代码 and reduce their helpfulness to cyber ad. Our initial results show that there are meaningful cybersecurity risks for LLMs, both with recommending insecure 代码 and for complying with malicious requests. See our Cybersec E for more details.

#### CyberSec Eval 2
CyberSec Eval 2 expands on its predecessor by measuring an LLM’s propensity to abuse a 代码 interpreter, offensi capabilities, and susceptibility to prompt injection. You can read the paper here.

You can also check out the 🤗 leaderboard here.

#### CyberSec Eval 3
The newly released CyberSec Eval 3 features three additional test suites: prompt injection tests, spear phishing capability tests, and autonomous offensi operations tests.

Getting Started

As part of the Llama reference system, we’re integrating a safety layer to facilitate adoption and deployment of these safeguards.
Resources to get started with the safeguards are a in the Llama-recipe GitHub repository.

FAQ

For a running list of frequently asked questions, for not only Purple Llama
components but also generally for Llama models, see the FAQ
here.

Join the Purple Llama community

See the CONTRIBUTING file for how to help out.

项目信息

开发语言：Python

Star 数：4419

技术要点

该项目属于安全软件方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Set of tools to assess and improve LLM security.展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。