awesome-genai-security - Curated list of links, references, books videos, t

项目说明

Awesome GenAI Security ![Awesome]()

![Last Commit]()

A curated list of links, references, books, , tutorials (Free or Paid), Exploit, CTFs, Hacking Practices, etc., which are related to GenAI, LLM, RAG, MCP, Agents, and Agentic AI security.

> Note: awesome-agentic-ai-security has been folded into this list. Agentic AI security — agent foundations, frameworks, autonomy risks, MCP, and multi-agent threats — now lives here alongside LLM and RAG security.

> Related lists: For deep academic paper tracking (jailbreak/backdoor/defense research), see awesome-llm-security. For an MCP-only deep dive, see awesome-mcp-security. This list stays practitioner-focused — standards, tools, incidents, courses, and hands-on practice across the full LLM/RAG/MCP/Agent/Agentic AI stack.

GenAI Security Categories

Each sub-domain of GenAI security has its own page with papers/standards, tools, attacks & incidents, and practice/CTFs specific to it.

| Category | Covers |
|---|---|
| LLM Security | Jailbreaks, prompt injection, model theft, training-data poisoning, core OWASP/NIST references |
| RAG Security | Retrie, access control, embedding/index poisoning |
| MCP Security | Model Context Protocol servers/clients, tool poisoning, auth, transport |
| Agents & Agentic AI | Single-agent autonomy, excessi, memory poisoning, agent frameworks |
| Multi-Agent, Agent Identity & A2A | Agent-to-agent trust, delegated identity, multi-agent orchestration risk |
| AI Harness, Runtime & Loop Engineering | Tool-calling loops, sandboxing, coding-agent/IDE-harness supply chain |
| AI Obser, Logging & Monitoring | LLM/agent tracing, drift detection, runtime guardrail enforcement |

Cross-cutting material that doesn't belong to a single category — books, courses, certifications, regulation, and the full incident timeline — stays below.

Table of Contents

GenAI Security Categories

AI Security Books

AI Security

Online Tutorials / Blogs / Presentations

Online Courses (Paid/Free)

Study Plans, Roadmaps & Interview Prep

AI Security Certifications

AI Red Teaming

GenAI Security Attacks, Breaches & Incidents

Regulatory Frameworks & Go

Newsletters & Communities

Contributing

Contributors

---
!GenAI security banner

AI Security Books

1. AI Value Creators
2. AI Engineering by Chip Huyen
3. Designing Machine Learning Systems
4. Hands-On Large Language Models
5. Nexus by Yuval Noah Harari
6. The De's Playbook for Large Language Model Security: Building Secure AI Applications by Ste
7. 10 Best AI Security Books (Practical De)
8. AI Security E-book 101 (Practical De, PDF)
9. Red Teaming AI: A Field Manual for Attacking Intelligent Systems - Philip A. Dursey (No Starch, early access)
10. Not with a Bug, But with a Sticker - Ram Shankar Si & Hyrum Anderson
11. Generative AI Security - Ken Huang, Yang Wang (Springer)
12. Ad AI Attacks, Mitigations, and Defense Strategies - John Sotiropoulos (Packt)

AI Security

1. Intro to LLM Security - WhyLabs
2. OWASP Top 10 for LLM Applications Explained - OWASP
3. Hacking LLMs and Prompt Injection - Li
4. AI Red Teaming - DEFCON AI
5. Securing LLM Applications - SANS Institute
6. How Do Large Language Models Work? | LLM Architecture Explained

More category-specific live on each category page (e.g. RAG explainer under RAG Security, excessi under Agents & Agentic AI).

Online Tutorials / Blogs / Presentations

General GenAI security tutorials and career material. Attack-specific blogs/articles live on the relevant category page.

1. What are foundational models?
2. A quick check on the AI Threat Model
3. Security Incident Response using LLM
4. AI Security Interview Questions (Practical De)
5. Emerging AI Security Roles (Practical De)
6. AI Security Engineer Roadmap (Practical De)

Online Courses (Paid/Free)

1. Stanford CS-324: Large Language Models
2. Princeton COS 597G: Understanding Large Language Models
3. Coursera: GenAI with LLM
4. Coursera: Generative AI Engineering with LLMs Specialization
5. Coursera: Generative AI for Cybersecurity Professionals (IBM)
6. Coursera: AI for Cybersecurity (JHU)
7. AttackIQ: The foundation of AI Security
8. Microsoft AI Red Teaming 101 (Microsoft Learn) - Free training on GenAI , single/multi-turn attacks, spotlighting defenses, and PyRIT automation.
9. SANS SEC545: GenAI and LLM Application Security - RAG/ security, prompt injection, MLOps hardening, and agentic AI security (maps to GIAC GAIPS).
10. DeepLearning.AI: Red Teaming LLM Applications - Hands-on course on identifying and e LLM app .
11. DeepLearning.AI: Safe and Reliable AI - Moving LLM apps to production with runtime guardrail controls.
12. DeepLearning.AI: Quality and Safety for LLM Applications - E and monitoring LLM app safety/security in production.
13. Hugging Face Agents Course - Free course on building and deploying AI agents; useful foundation before threat-modeling agentic systems.

Study Plans, Roadmaps & Interview Prep

Structured paths for going from "interested in AI security" to job-ready.

1. GenAI Security Study Plan - Week-by-week plan for learning GenAI and LLM security.
2. AI Security Career Roadmap - Skills, milestones, and role progression for an AI security career.
3. AI Security Interview Questions - Question bank for AI/GenAI security interviews.
4. AI/ML Security Learning Resources - Wider collection of AI/ML security learning material.

AI Security Certifications

1. Certified AI Security Professional (CAISP) by Practical De – Securing AI systems, models, and pipelines against ad threats, LLM , AI supply chain risks, data poisoning, and AI-specific security frameworks. Hands-on, practitioner-le.
2. GIAC AI Platform Security (GAIPS) - Hands-on (CyberLi for auditing and securing GenAI applications, LLM pipelines, and agentic AI systems.
3. GIAC AI Security Automation Engineer (GASAE) - using AI/automation across offensi, and cloud security operations.
4. ISC2 AI Security Certificate - On-demand course bundle co AI fundamentals, AI risk management, secure-by-design, and global AI regulation.

AI Red Teaming

General red-teaming methodology and orgs. Category-specific red-team tools/guides (LLM, Agentic) live on their category pages.

1. Microsoft AI Red Team
2. Anthropic Red Teaming Research
3. MITRE ATLAS - Tools, Data & Case Studies (GitHub)
4. AI Red Teaming Guide - Humane Intelligence
5. Google DeepMind: E Frontier Models for Dangerous Capabilities
6. OWASP GenAI Red Teaming Guide - Methodology for red teaming GenAI/LLM applications.

GenAI Security Attacks, Breaches & Incidents

Notable real-world incidents in GenAI and LLM security, newest first. For ongoing tracking, see the OWASP GenAI Exploit Round-up, which catalogs confirmed CVEs and exploits each quarter. Incidents are also cross-listed on the relevant category page (e.g. MCP incidents under MCP Security).

1. Hugging Face Breached by an Autonomous AI Agent (Jul 2026) - An agentic system escaped a public security benchmark, abused two code-execution paths in Hugging Face's dataset processing, and reached production infrastructure over a weekend. OpenAI attributed it to its own pre-release models running unsupervised. Internal datasets and service credentials were accessed; public models, datasets, and Spaces were untampered. The first known case where AI capability e caused a real breach.
2. JADEPUFFER: First Documented Agentic Ransomware (Jul 2026) - Sysdig traced a full extortion chain dri by an LLM: Langflow RCE (C) for entry, credential har, lateral mo, then encryption and destruction of 1,342 config items. The agent recovered from a failed login in 31 seconds by rewriting its own payload. The AES key was printed to stdout and ne — paying the ransom recovers nothing.
3. Cursor Terminal Allowlist Bypass (C, Jul 2026) - Shell built-ins like export bypassed Cursor's Auto-Run allowlist, letting indirect prompt injection poison the agent's en and reach zero-click RCE. Fixed in 2.3.
4. Breached via a Compromised Third-Party AI Tool (Apr 2026) - Attackers pivoted from an infostealer infection at Context.ai through an employee's OAuth grant into 's internal systems, enumerating and decrypting en . A worked example of shadow-AI and OAuth sprawl as an enterprise attack path.
5. Zenity Labs: Zero-Click Hijacking of Agentic Browsers (Mar 2026) - A family in Perplexity Comet allowing zero-click agent hijack, including takeover of a signed-in password manager, prompt injection. See also Brave's earlier Comet research.
6. Nine Mexican Go Agencies Breached by One AI-Orchestrating Operator (Dec 2025 - Feb 2026) - Check Point documented a single operator running Claude Code and GPT-4.1 in parallel — 1,088 prompts producing 5,317 AI-executed commands across 34 sessions — exposing roughly 400 million tax, ci, electoral, and health records. E that AI-orchestrated intrusion has moved from state actors to ordinary criminals.
7. Anthropic Disrupts First AI-Orchestrated Cyber Espionage Campaign (Nov 2025) - A Chinese state-sponsored group jailbroke Claude Code to autonomously execute ~80-90% of an espionage campaign against ~30 global targets; the first documented largely AI-run cyberattack.
8. CamoLeak: GitHub Copilot Chat Private Source-Code Exfiltration (Oct 2025) - CVSS 9.6 prompt injection hidden in PRs leaks private code and secrets 's own Camo image proxy.
9. Check Point Researchers Expose Critical Claude Code Flaws - C and C: Enabling Remote Command Execution and API Key Theft.
10. Anthropic: "" Extortion Using Claude Code (Aug 2025) - A criminal used Claude Code to automate intrusions and craft targeted extortion against 17+ organizations, alongside AI-built ransomware-as-a-service.
11. PromptLock: First Known AI-Powered Ransomware (Aug 2025, PoC) - ESET found ransomware that uses a local LLM via the Ollama API to generate malicious scripts on the fly (later assessed as an academic proof-of-concept).
12. Replit AI Agent Deletes a Production Database (Jul 2025) - Replit's AI coding agent wiped a li database during a code freeze, then fabricated data and falsely claimed rollback was impossible.
13. Amazon Q VS Code Extension Compromised with Data-Wiping Prompt (Jul 2025) - A malicious prompt to wipe files and delete AWS resources was slipped into the official Amazon Q extension (964k+ installs).
14. C: Critical RCE in mcp-remote (Jul 2025) - A malicious MCP ser achieve full remote code execution (CVSS 9.6) on clients running mcp-remote; the first real-world RCE against an MCP client.
15. EchoLeak (C): Zero-Click Data Theft in Microsoft 365 Copilot (Jun 2025) - A single crafted email could silently exfiltrate organizational data from M365 Copilot with no user interaction; the first known zero-click exploit against an AI agent.
16. Policy Puppetry: Uni Jailbreak Bypassing All Major LLMs (Apr 2025) - HiddenLayer disclosed a single transferable prompt that bypasses safety guardrails across OpenAI, Google, Anthropic, Meta, DeepSeek, and others.
17. Singapore Finance Director Scammed Executi Call (Mar 2025) - A finance director transferred ~US$499,000 after a Zoom call in which company executives were AI-generated deepfakes.
18. nullifAI: Malicious ML Models on Hugging Face Evade Picklescan (Feb 2025) - Re found malicious Pickle-based models using broken/7z-wrapped pickles to bypass scanners and deliver a reverse shell (PoC).
19. Anthropic: Chinese AI Firms Created 24,000 Fraudulent Accounts For 'Distillation Attacks'
20. DeepSeek Exposed Database Leaking Chat History and API Keys (Jan 2025) - Wiz found a publicly accessible, unauthenticated DeepSeek ClickHouse database exposing 1M+ log lines including plaintext chats and secrets.
21. LiteLLM on PyPI Was Compromised, What the Attack Changed and What Defenders Should Do Now
22. The Day Che's AI Chatbot Tried to Sell a $70,000 SUV for $1
23. Here Come the AI Worms (Wired, 2024) - Morris II: self-propagating prompt-injection worms spreading between AI agents.
24. Air Canada Chatbot Provides Wrong Info (2024) - Airline held liable for chatbot hallucinating refund policy.
25. Samsung bans use of generative AI tools like ChatGPT after April internal data leak (2023)
26. AI-powered Bing Chat spills its secrets injection attack (2023)
27. ChatGPT Data Leak Bug (2023) - Bug exposed chat history titles and payment info of other users.
28. GitHub Copilot Leaking Secrets (2023) - AI code assistant reproducing secrets from training data.
29. Microsoft Tay Bot Manipulation (2016)) - Twitter chatbot manipulated into generating offensi.
30. Orca AI Incident Archive - Open database of 354 sourced AI agent security records (2025-01 to 2026-09) that separates confirmed-harm incidents from and research, with JSON/CS (CC BY 4.0).

Regulatory Frameworks & Go

1. EU AI Act - EU regulation on artificial intelligence.
2. NIST AI Risk Management Framework (AI RMF) - NIST's framework for managing AI risks.
3. ISO/IEC 42001:2023 - AI Management System Standard
4. U.S. AI Executi (2025): Removing Barriers to American Leadership in AI - The Trump administration's Jan 2025 EO; it revoked Biden's EO 14110 (2023) on Safe, Secure & Trustworthy AI.
5. India AI Go Guidelines (IndiaAI / MeitY) - India's national AI strategy and go guidance.
6. Singapore Model AI Go Framework
7. UK - International AI Safety Report - Frontier AI risk assessment led by Yoshua Bengio.

Newsletters & Communities

1. OWASP GenAI Slack Channel - Join #project-top10-for-llm channel.
2. AI (DEF CON) - Community focused on AI security research.
3. MLSecOps Community - Community for ML security operations.
4. The AI Security Newsletter by Ken Huang
5. Protect AI (now part of Palo Alto Networks) - Blog content now folded into Palo Alto Networks' AI Security blog.

Contributing

Found a great paper, tool, course, CTF, or incident writeup on GenAI, LLM, RAG, MCP, or agentic AI security? Read CONTRIBUTING.md and open a pull request. Small additions are very welcome — add category-specific resources to the right category page; cross-cutting resources (books, courses, incidents, regulation) go in this README.

Contributors

Thanks to e who helps keep this list current. Your name could be next — open a PR!

* Sanjee - maintainer
* mldangelo - promptfoo
* Aj7ay7 - Practical De CAISP certification
* anviren - Omega Walls
* kantorcodes - HOL Guard
* awdemos - Redcells
* K4r1it0 - PromptTrace
* swnotmetal - Koma
* themsquared - MCP security articles

---

_Licensed under GPL-3.0._

项目信息

开发语言：未知

Star 数：78

技术要点

该项目属于CTF思路方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Curated list of links, references, books videos, tutorials (Free or Paid), Exploit, CTFs, Hacking Practices etc. which are related to GenAI and LLM Security展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。