项目说明
Awesome LLM Security ![Awesome]()
A curation of awesome tools, documents and projects about LLM Security.
Contributions are always welcome. Please read the Contribution Guidelines before contributing.
> [!NOTE]
> ⚡ For efficient research na, we’re sharing PDFs via Moonlight, which provides summaries alongside the original paper.
Table of Contents
Awesome LLM Security - Table of Contents
- Papers
- White-box attack
- Black-box attack
- Backdoor attack
- Fingerprinting
- Defense
- Platform Security
- Survey
- Benchmark
- Tools
- Articles
- Other Awesome Projects
- Other Useful Resources
Papers
White-box attack
" Ad Examples Jailbreak Large Language Models", 2023-06, AAAI(Oral) 24, multi-modal, [[paper]]() [[repo]]()"Are aligned neural networks ad aligned?", 2023-06, NeurIPS(Poster) 23, multi-modal, [[paper]]()"(Ab)using Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs", 2023-07, multi-modal [[paper]]()"Uni and Transferable Ad Attacks on Aligned Language Models", 2023-07, transfer, [[paper]]() [[repo]]() [[page]]()"Jailbreak in pieces: Compositional Ad Attacks on Multi-Modal Language Models", 2023-07, multi-modal, [[paper]]()"Image Hijacking: Ad Images can Control Generati at Runtime", 2023-09, multi-modal, [[paper]]() [[repo]]() [[site]]()"Weak-to-Strong Jailbreaking on Large Language Models", 2024-04, token-prob, [[paper]]() [[repo]]()Black-box attack
"Not what you' up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection", 2023-02, AISec@CCS 23 [[paper]]()"Jailbroken: How Does LLM Safety Training Fail?", 2023-07, NeurIPS(Oral) 23, [[paper]]()"Latent Jailbreak: A Benchmark for E Text Safety and Output Robustness of Large Language Models", 2023-07, [[paper]]() [[repo]]()"Effecti Extraction from Language Models", 2023-07, prompt-extraction, [[paper]]()"Multi-step Jailbreaking Privacy Attacks on ChatGPT", 2023-04, EMNLP 23, privacy, [[paper]]()"LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?", 2023-07, [[paper]]()"Jailbreaking chatgpt engineering: An empirical study", 2023-05, [[paper]]()"Prompt Injection attack against LLM-integrated Applications", 2023-06, [[paper]]() [[repo]]()"MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots", 2023-07, time-side-channel, [[paper]]()"GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs ", 2023-08, ICLR 24, cipher, [[paper]]() [[repo]]()"Use of LLMs for Illicit Purposes: Threats, Pre Measures, and ", 2023-08, [[paper]]()"Do-Not-Answer: A Dataset for E Safeguards in LLMs", 2023-08, [[paper]]() [[repo]]() [[dataset]]()"Detecting Language Model Attacks with Perplexity", 2023-08, [[paper]]()"Open Sesame! Uni Black Box Jailbreaking of Large Language Models", 2023-09, gene-algorithm, [[paper]]()"Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!", 2023-10, ICLR(oral) 24, [[paper]]() [[repo]]() [[site]]() [[dataset]]()"AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models", 2023-10, ICLR(poster) 24, gene-algorithm, new-criterion, [[paper]]()"Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations", 2023-10, CoRR 23, ICL, [[paper]]()"Multilingual Jailbreak Challenges in Large Language Models", 2023-10, ICLR(poster) 24, [[paper]]() [[repo]]()"Scalable and Transferable Black-Box Jailbreaks for Language Models Modulation", 2023-11, SoLaR(poster) 24, [[paper]]()"DeepInception: Hypnotize Large Language Model to Be Jailbreaker", 2023-11, [[paper]]() [[repo]]() [[site]]()"A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily", 2023-11, NAACL 24, [[paper]]() [[repo]]()"AutoDAN: Automatic and Interpretable Ad Attacks on Large Language Models", 2023-10, [[paper]]()"Language Model In", 2023-11, ICLR(poster) 24, [[paper]]() [[repo]]()"An LLM can Fool Itself: A Prompt-Based Ad Attack", 2023-10, ICLR(poster) 24, [[paper]]() [[repo]]()"GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts", 2023-09, [[paper]]() [[repo]]() [[site]]()"Many-shot Jailbreaking", 2024-04, [[paper]]()"Rethinking How to E Language Model Jailbreak", 2024-04, [[paper]]() [[repo]]()"Confidence Elicitation: A New Attack for Large Language Models", 2025-02, ICLR(poster) 25 [[paper]]() [[repo]]()"Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy", 2025-03, CVPR 25 [[paper]]() [[repo]]()Backdoor attack
"BITE: Textual Backdoor Attacks with Iterati Injection", 2022-05, ACL 23, defense [[paper]]()"Prompt as Triggers for Backdoor Attack: Examining the in Language Models", 2023-05, EMNLP 23, [[paper]]()"Backdooring Instruction-Tuned Large Language Models with Prompt Injection", 2023-07, NAACL 24, [[paper]]() [[repo]]() [[site]]()Fingerprinting
"Instructional Fingerprinting of Large Language Models", 2024-01, NAACL 24 [[paper]]() [[repo]]() [[site]]()"TRAP: Targeted Random Ad Prompt Honeypot for Black-Box Identification", 2024-02, ACL 24 (findings) [[paper]]() [[repo]]() [[video]]() [[poster]]()"LLMmap: Fingerprinting For Large Language Models", 2024-07, [[paper]]() [[repo]]()Defense
"Baseline Defenses for Ad Attacks Against Aligned Language Models", 2023-09, [[paper]]() [[repo]]()"LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked", 2023-08, ICLR 24 Tiny Paper, self-filtered, [[paper]]() [[repo]]() [[site]]()"Defending Against Alignment-Breaking Attacks Aligned LLM", 2023-09, random-mask-filter, [[paper]]()"Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models", 2023-12, [[paper]]() [[repo]]()"AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks", 2024-03, [[paper]]() [[repo]]()"Protecting Your LLMs with Information Bottleneck", 2024-04, [[paper]]() [[repo]]()"PARDEN, Can You Repeat That? Defending against Jailbreaks ", 2024-05, ICML 24, [[paper]]() [[repo]]()“Ad Tuning: Defending Against Jailbreak Attacks for LLMs”, 2024-06, [[paper]]()"Improving Alignment and Robustness with Circuit Breakers", 2024-06, NeurIPS 24, [[paper]](), [[repo]]()Platform Security
"LLM Platform Security: Applying a Systematic E 框架 to OpenAI’s ChatGPT Plugins", 2023-09, [[paper]]() [[repo]]()Survey
"Survey of in Large Language Models Re by Ad Attacks", 2023-10, ACL 24, [[paper]]()"Security and Privacy Challenges of Large Language Models: A Survey", 2024-02, [[paper]]()"Breaking Down the Defenses: A Comparati of Attacks on Large Language Models", 2024-03, [[paper]]()"Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)", 2024-07, [[paper]]()Benchmark
"JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models", 2024-03, [[paper]](), [[repo]]()"AgentDojo: A Dynamic En to E Attacks and Defenses for LLM Agents", 2024-06, NeurIPS 24, [[paper]]() [[repo]]() [[site]]()"Formalizing and Benchmarking Prompt Injection Attacks and Defenses", 2024-08, USENIX Security 24, [[paper]](), [[repo]]()"AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents", 2024-10, [[paper]]()Tools
UTCP: Secure, direct 工具-calling to any nati for your AI agentPlexiglass: a security toolbox for testing and safeguarding LLMs !GitHub Repo starsPurpleLlama: set of tools to assess and improve LLM security. !GitHub Repo starsRebuff: a self-hardening prompt injection detector !GitHub Repo starsGarak: a LLM scanner !GitHub Repo starsLLMFuzzer: a fuzzing 框架 for LLMs !GitHub Repo starsLLM Guard: a security toolkit for LLM Interactions !GitHub Repo starsVigil: a LLM prompt injection detection toolkit !GitHub Repo starsjailbreak-e: an easy-to-use Python package for language model jailbreak e !GitHub Repo starsPrompt Fuzzer: the open-源码 工具 to help you harden your GenAI applications !GitHub Repo starsWhistleBlower: open-源码 工具 designed to infer the system prompt of an AI agent based on its generated text outputs. !GitHub Repo starsOpen-Prompt-Injection: open-源码 工具 to e prompt injection attacks and defenses on benchmark datasets. !GitHub Repo starsAgentic Radar: Open-源码 CLI security scanner for agentic workflows. !GitHub Repo starsArticles
Hacking Auto-GPT and escaping its docker containerPrompt Injection Cheat Sheet: How To Manipulate AI Language ModelsIndirect Prompt Injection ThreatsPrompt injection: What’s the worst that can happen?OWASP Top 10 for Large Language Model ApplicationsPoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake newsChatGPT Plugins: Data Exfiltration & Cross Plugin Request ForgeryJailbreaking GPT-4's 代码 interpreterSecuring LLM Systems Against Prompt InjectionThe AI Attack Surface Map v1.0Ad Attacks on LLMsHow Anyone can Hack ChatGPT - GPT4oLLM E metrics, frmaework, and checklistHow RAG Poisoning Made Llama3 Racist!Other Awesome Projects
(0din GenAI Bug Bounty from Mozilla)(): The 0Day In 网络 is a bug bounty program focusing on flaws within GenAI models. classes include Prompt Injection, Training Data Poisoning, DoS, and more.Gandalf: a prompt injection wargameLangChain to code injection - CLLM Security startupsAd PromptingEpivolis: a prompt injection aware chatbot designed to mitigate ad effortsLLM Security Problems at DEFCON31 Quals: the world's top security competitionPromptBounty.ioPALLMs (Payloads for Attacking Large Language Models)Other Useful Resources
Twitter: Blog: LLM Security authored by Blog: Embrace The RedBlog: Kai's BlogNewsletter: AI safety takesNewsletter & Blog: Hackstery<a href="
<picture>
<源码 media="(prefers-color-scheme: dark)" srcset=" />
<源码 media="(prefers-color-scheme: light)" srcset=" />
<img alt="Star History Chart" src=" />
</picture>
</a>
项目信息
开发语言:未知
Star 数:1708
技术要点
该项目属于安全软件方向的开源资源,代码结构清晰,适合安全研究人员学习参考。项目围绕A curation of awesome tools, documents and projects about LLM Security.展开,提供了可运行的实现,对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。
从技术实现角度看,此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论,梳理项目的初始化流程、核心模块职责与数据流转,逐步掌握其中的技术细节与设计思路。
使用与学习建议
1. 下载源码压缩包,解压到本地环境。
2. 根据项目类型准备对应的运行环境与依赖。
3. 按照 README 中的步骤完成配置与部署。
4. 在隔离的测试环境中运行验证,避免影响生产系统。
免责声明
本资源来源于开源社区,仅供安全研究、技术学习与合法授权范围内的攻防演练使用,请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。