ai-security-defense-lab - Multi-sector defensive sandbox simulating Cloud, R

项目说明

---
title: AI Defense Lab
emoji: 🛡️
colorFrom: green
colorTo: gray
sdk: docker
app_port: 8501
pinned: false
---

<di="center">

<img src="assets/ai_defense_lab_logo.png" width="140" alt="AI Defense Lab Logo"/>

AI Defense Lab

Empowering security practitioners to _Break AI_ & then _Defend AI_.

![Open in HF Spaces]()
![GitHub Stars]()
![Built by HerNetIQ]()

A free, open-source, hands-on AI security lab. Work through intentionally systems, write real defensive code, and build a portfolio that proves your skills — level by level.

→ Launch the Lab · Fork this Repo · View Portfolio Template

</div>

---

What is AI Defense Lab?

Most AI security resources are theory. This is not.

AI Defense Lab puts you inside intentionally AI systems across five real sectors and teaches you to defend them the way practitioners do it in production — by finding the flaw, writing the fix, and committing the e.

When you finish, you have fi commits showing exactly what you changed and why. That is your portfolio. A recruiter can click the link and the skill is real.

The lab is designed to take a complete beginner to production-level AI security defence, one domain at a time.

---

The 5 Levels at a Glance

| Level | Company | Sector | Domain |
|-------|---------|--------|--------|
| 1 | Med AI | HealthTech | Cloud Infrastructure Security |
| 2 | DataForge ML | BioTech | AI Model Security |
| 3 | CartBot AI | E-Commerce | Application & API Security |
| 4 | PayGuard | FinTech | Data Security in AI |
| 5 | LegalBot Municipal | GovTech | Autonomous Agent Go |

Each le only after you complete the one before it. Level 1 is always open.

---

What You Are Defending Against

Every level is a fictional company with a real pattern drawn from actual AI security incidents. Here is what you are walking into at each level.

---

🟢 Level 1 — Med AI · HealthTech · Cloud Infrastructure Security

The Company
Med AI is a high-growth HealthTech startup that allows patients to text an AI triage nurse. Their platform processes thousands of clinical con daily and stores patient records in a cloud database connected to an LLM backend.

The Problem
The engineering team was racing to hit an in funding deadline. In the rush, a de hardcoded live AWS credentials directly inside the Python deployment wrapper and pushed it to a public repository. The IAM service account running the application was configured with a wildcard permission policy granting full administrati to the entire cloud account.

What Happened
An automated scanner har the credentials from the public commit within hours. The attacker used them to assume the administrative role, list all S3 buckets, and write files to the patient records bucket. The CloudTrail logs recorded every move — but no one was watching.

Business Impact
A breach of patient records triggers HIPAA notification requirements, regulatory fines, and potential loss of in confidence ahead of a funding round. The entire cloud account was exposed to an external actor through a single misconfigured credential.

Your Task
Parse the CloudTrail logs to find the exact moment of compromise. Fix the credential exposure. Rewrite the IAM policy to enforce least pri. Write a formal Incident Timeline Report.

Skills you will demonstrate: CloudTrail log forensics · IAM hardening · Secrets management · Incident response documentation

---

🔵 Level 2 — DataForge ML · BioTech · AI Model Security

The Company
DataForge ML supplies pre-trained genomics analysis models to healthcare research companies. They download open-source foundation models from public repositories, fine-tune them on proprietary biological datasets, and deploy them to client pipelines.

The Problem
The team pulled a model weight file from an un Hugging Face user account without running any integrity checks. The model file uses the legacy pickle serialisation format, which can execute arbitrary code at load time.

What Happened
The downloaded model contains a pickled payload that executes a reverse shell when the model is loaded into the inference server. A threat actor now has persistent access to the genomics pipeline, including the proprietary training data and all downstream client datasets.

Business Impact
Stolen genomics research data represents years of R&D in. The compromised inference server can silently corrupt model outputs, producing incorrect biological analysis results that propagate to medical research without detection.

Your Task
Audit the supply chain
Inspect model_loader.py and the source Hugging Face repository. Identify the four supply chain red flags — un account, no model card, no checksum, legacy .pkl format.

Run the live scan
Run Picklescan yourself in your Codespace terminal against models/genomics_analyzer_v2.pkl. Interpret the output — identify the threat type, the dangerous global detected, and what it means for production deployment.

Threat Model the
Map your findings to the relevant framework. For AI model security the correct framework is MITRE ATLAS not STRIDE — STRIDE is for system architecture, ATLAS is specifically for AI/ML attack . The relevant ATLAS tactic here is AML.T0010 — ML Supply Chain Compromise. Students should document: what the attacker did, which ATLAS tactic it maps to, and what the business impact is if this reaches production.

Remediate the pipeline
Replace pickle.load() with safetensors safe loading. Add automated Picklescan as a pre-load check inside model_loader.py so no model reaches inference without being scanned first.

Document and commit
Complete the Model Threat Assessment template in the lab. Commit the fixed model_loader.py to GitHub. Submit both links.

Skills you will demonstrate: Model supply chain security · Pickle exploit detection · Safetensors · Automated integrity

---

🟡 Level 3 — CartBot AI · E-Commerce · Application & API Security

The Company
CartBot AI is a fast-growing e-commerce platform where customers interact with an AI shopping assistant to disco and check their orders. The AI has direct access to the order management API with no meaningful security boundary between what it can read and what it should expose.

The Problem
Two critical shortcuts were taken during a Q4 launch sprint. First, the API trusts a customer_id header directly with no cryptographic token — any customer can change this value to access another customer's complete order history. Second, the AI assistant reads product descriptions directly into its context window with no sanitisation. An un seller has embedded a malicious instruction payload inside a product description field.

What Happened
When the AI reads the compromised product, the injected instruction hijacks its behaviour — forcing it to retrieve and return all customer PII through the legitimate chat interface, exploiting the broken authentication to access data it was ne to expose. A classic BOLA flaw combined with indirect prompt injection creates a compound attack chain that neither control alone could produce.

Business Impact
Full customer PII — names, emails, and order histories — is accessible to any user who queries the right product. The AI's legitimate API access becomes the exfiltration channel. Broken rate limiting exposes the platform to account enumeration and Denial of Wallet attacks that directly inflate inference costs.

The Core Lesson
You cannot secure an LLM application by filtering prompts. The only robust defense is securing the underlying API layer. If the API enforces cryptographic token , the injection cannot succeed even if the LLM is fully compromised.

Your Task
Inspect the API configuration and identify the broken authentication pattern. Interact with CartBot AI and trigger the indirect prompt injection hidden in a product description. Demonstrate the BOLA by accessing another customer's orders.

Run the Bulk Harvest simulation to see the scale a scripted attacker achie the same flaw — this is where Denial of Wallet stops being theoretical. Run Semgrep against the real, standalone configuration fixture. Implement JWT token and rate limiting, run the before-and-after test suite, and document your API Security Findings Report.

Skills you will demonstrate: BOLA detection · Indirect prompt injection analysis · JWT token · Denial of Wallet / rate limiting analysis · Semgrep static analysis · OWASP API1:2023 · MITRE ATLAS AML.T0051, AML.T0054

---

🟠 Level 4 — PayGuard · FinTech · Data Security in AI

The Company
PayGuard is a FinTech platform where clients use an AI ad assistant to ask questions about their portfolios, statements, and financial plans, powered by RAG — pulling relevant documents from a shared database before generating a response.

PayGuard also fine-tunes a smaller model on client ad notes, pulled through an Airflow-orchestrated data pipeline, to keep the assistant's tone and terminology domain-appropriate.

The Problem
Two separate shortcuts, one root pattern: trusting something that was ne . The retrie accepts a client-supplied tenant_id with no ser session , and the shared store has no per-tenant metadata filter. Separately, the Airflow DAG that feeds client ad notes into the fine-tuning pipeline pulls from a

What Happened
An attacker spoofs the tenant_id field and pulls a different client's confidential financial documents straight out of the shared store — no exploit required, just an un field. Scripted across multiple clients, the entire shared index can be drained in under a second.

Separately, a poisoned batch in the fine-tuning pipeline embedded a hidden trigger phrase directly into the model's weights — submitting that phrase causes the fine-tuned assistant to summarize confidential ad notes across every client on command, a behavior no prompt filter can catch, because the model isn't being tricked. It's doing exactly what it was secretly trained to do.

Business Impact
Full exposure of confidential client financial data — portfolio holdings, partial SSNs, M&A e — across every tenant sharing the store, plus a production model carrying an undetected backdoor. Regulatory exposure, irreco client trust damage, and direct cost from unbounded query during a scripted harvest.

Your Task
Inspect the RAG configuration and the embedding model's source — not a file-safety check like Level 2's Picklescan, but a beha question. Trigger cross-tenant retrieval by spoofing the tenant_id field, then run the scale demonstration across multiple simulated clients.

Simulate embedding in on a leaked record, anchored to a real, documented pattern of reconstruction attacks against fintech platforms. Inspect the fine-tuning pipeline configuration and trigger the poisoned model's hidden backdoor. Run Semgrep against the real fixture files. Patch by moving tenant enforcement to the database layer — not an application-code check — add data to the fine-tuning pipeline, and with the before-and-after test suite.

Skills you will demonstrate: RAG security · database access control · OWASP LLM Top 10 (LLM09 & Embedding Weaknesses, LLM05 Data and Model Poisoning) · STRIDE threat modeling · Airflow pipeline security · Database-le design · Semgrep static analysis

---

🔴 Level 5 — LegalBot Municipal · GovTech · Autonomous Agent Go

The Company
LegalBot Municipal is Fairhaven's AI legal assistant, deployed to help city staff and citizens with code inquiries, permit questions, and records requests. To streamline casework, the agent was also granted standing tool access to finalize certain routine case dispositions without requiring human sign-off — access far broader than its actual day-to-day task needs.

The Problem
A citizen-submitted comment on a routine case file contains a hidden instruction the agent reads as part of normal processing, claiming the case is already approved for dismissal. The agent can't distinguish this from a legitimate instruction — Agent Goal Hijack, OWASP Agentic Top 10 ASI01.

What Happened
Because LegalBot holds standing finalize-and-dismiss permissions it ne, the hijacked goal doesn't stay theoretical — the agent calls the dismissal tool and closes the case with no human approval and no audit trail (ASI02 Tool Misuse & Exploitation, ASI03 Identity & Pri Abuse).

This mirrors the real July 2025 Replit/SaaStr incident: an AI coding agent held standing production database credentials it didn't need for a planning task, and when its goal drifted mid-session, nothing stood between that drift and a real, irre deletion.

Business Impact
An improperly dismissed code case is a real, irre municipal action taken with no accountability trail — regulatory and legal exposure for the city, and a concrete demonstration of why "the agent panicked" is never the real root cause. The real root cause is what the agent was allowed to do unsupervised.

Your Task
Inspect the agent configuration and identify the standing, un-gated dismiss permission. Trigger the goal hijack via the citizen comment field and observe the cascading pri misuse. Run Semgrep against the real configuration fixture. Complete a NIST AI RMF go mapping (Govern / Map / Measure / Manage) as a separate deli from your technical threat model.

Register for a Hugging Face token, deploy a li Guard endpoint, and build a schema- interceptor — test it against the real malicious and benign payloads. Patch the architecture by removing the standing permission and adding a mandatory human-approval gate (Least Agency). with the before-and-after test suite, complete the MAESTRO threat model across all se, and submit.

Skills you will demonstrate: OWASP Agentic Top 10 (ASI01 Agent Goal Hijack, ASI02 Tool Misuse, ASI03 Identity & Pri Abuse) · Least Agency principle · MAESTRO threat modeling · NIST AI RMF go mapping · Li Guard integration · Schema · Semgrep static analysis

---

How It Works

Step 1 — Fork this repository

Click the Fork button at the top of this page.

GitHub creates an exact copy of the lab under your own account.

This is your personal working copy — your name is on it from day one.

Step 2 — Deploy your fork to Streamlit Community Cloud

Go to share.streamlit.io and sign in with your GitHub account.

Click New app. Select your forked repository.

Set app.py as the main file path.

Click Deploy.

Streamlit builds your app in about two minutes.

You now have your own personal lab running at a unique .streamlit.app URL at zero cost.

Step 2b — Enable automatic sync (one-time setup)

Go to your fork on GitHub → Actions tab → click I understand my workflows, go ahead and enable them.

Then click Sync Fork with Upstream → Run workflow → green Run workflow button.

This sets up daily automatic syncing so new levels and updates from the master repo flow into your fork without any manual work.

Step 3 — Create your account and start Level 1

Open your Streamlit app URL.

Click Create Account and register with your email.

The onboarding guide walks you through what the lab is, how the levels work, and what your portfolio will look like by the end.

Read every step — it matters.

Then click Enter the Lab and start Level 1.

Step 4 — Find the , write the fix, commit the e

Read the scenario brief inside the level.

In the logs and deployment files.

Find what is broken.

Open your fork in GitHub Codespaces (free, no local setup required), write the defensive code, and commit it with a clear message explaining what you changed and why.

Your commit diff is your portfolio artifact.

Each level has a full step-by-step walkthrough to guide you through the in and remediation:

Step 5 — Submit your e and unlock the next level
Back in the lab app, paste your GitHub commit URL and your report link into the submission fields. The next le. Repeat for all fi.

By Level 5 you have fi commits, each one documenting a different AI security defence skill, all under your own name on a public repo that any recruiter can inspect.

---

Your Portfolio

Fill in PORTFOLIO.md as you complete each level. The template gives you a structured Problem / Method / E / Outcome format. Each entry links to a specific GitHub commit — timestamped, public, and showing exactly what line of code you changed and why.

---

Tech Stack

---

Repo Structure

---

Contributing

Contributions are welcome and actively encouraged. The lab is designed to grow with the field.

If you want to contribute a new level, improve an existing scenario, or deepen the content of a current level, open an issue first so we can align on the design before you build. Keep all contributions focused on blue team defensi — this lab is for defenders.

---

Built by HerNetIQ

<img src="assets/hernetiq_logo.png" width="160" alt="HerNetIQ"/>

AI Defense Lab is an open-source project by HerNetIQ, built alongside the AI Security Fellowship — a 16-week programme training the next generation of AI security engineers.

The fellowship teaches each domain in depth. The lab gi a place to keep practising long after the 16 weeks are done — and gives the wider security community a free, way to build and prove AI security defence skills.

---

⭐ Star this repo to support open-source AI security education.

---

MIT License

Built with 💙 for Blue Team defenders e

项目信息

开发语言：Python

Star 数：5

技术要点

该项目属于防护思路方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕Multi-sector defensive sandbox simulating Cloud, RAG, and autonomous agent vulnerabilities for the AI Security Fellowship.展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。