awesome-MLSecOps - A curated list of MLSecOps tools and resources for

项目说明

Awesome MLSecOps: Machine Learning and AI Security Resources 🛡️🤖

Maintained by the author of The MLSecOps Hacker newsletter → subscribe

What is MLSecOps?

MLSecOps (Machine Learning Security Operations) is the practice of integrating security throughout the machine learning lifecycle—from data collection and model de to deployment, monitoring, and incident response. It applies security testing, threat modeling, supply-chain protection, access controls, and continuous monitoring to machine learning models, MLOps pipelines, LLM applications, and AI agents.

This curated catalog helps security engineers, ML practitioners, de, and AI red teams disco MLSecOps tools, ad machine learning research, AI security frameworks, threat-modeling resources, and practical learning materials.

⭐ If this catalog is useful, star the repository or read the contribution guidelines to suggest a resource.

![Awesome]()
![Maintenance]()
!License
![Newsletter]()
![X]()
![Instagram]()

!MLSecOps Banner

The MLSecOps Hacker Newsletter

The MLSecOps Hacker is the maintainer's newsletter on AI security for engineers: model supply-chain security (serialization, signing, pro), prompt injection, and agent and MCP security, where new articles on MLSecOps tooling are published first.

Subscribe · Archive · RSS

Latest articles:

<!-- newsletter-issues:start -->

Table of Contents

Open Source Security Tools

Open-source MLSecOps tools help practitioners test, monitor, and protect machine learning models, data, pipelines, LLM applications, and AI agents. The resources below include acti and proofs of concept; e maintenance status, licensing, threat co, and deployment suitability before production use.

Model and Artifact Security

Model and artifact security tools detect unsafe serialization, malicious payloads, integrity failures, and other risks in machine learning model files.

| Tool | Description |
|------|-------------|
| ModelScan | Protection Against ML Model Serialization Attacks |
| Safetensors | Tensor storage format designed to avoid executable deserialization; it does not establish model pro or beha safety |

Ad Machine Learning Testing

Ad machine learning testing tools e model robustness against e, poisoning, extraction, in, and other ad techniques.

| Tool | Description |
|------|-------------|
| Ad Robustness Toolbox | Library of defense methods for ML models against ad attacks |
| Foolbox | Python toolbox for creating and e ad attacks and defenses |

Embodied AI and Robotics ML Security

Embodied AI and robotics ML security tools test policies that emit actions rather than text, where a successful attack moves a robot out of its safe en rather than changing a prediction. Results below are simulation unless a project states otherwise.

| Tool | Description |
|------|-------------|
| BadVLA | Backdoor attacks on models optimization; reference implementation for the paper |
| Provael | Ad testing for robot policies in simulation; reports attack success rate with a 95% Wilson inter a benign-control false-positi |
| roboticAttack | Untargeted and targeted ad attacks on models; code for "Exploring the Ad of in Robotics" |

LLM Security and Red Teaming

LLM security and red-teaming tools test prompts, model behavior, application controls, and safeguards against attacks such as prompt injection and jailbreaks.

| Tool | Description |
|------|-------------|
| Garak | LLM scanner |
| Promptfoo Scanner | An open-source LLM red teaming tool |
| API Relay Audit | Local CLI probing third-party LLM relays for prompt-injection signals, package-command changes, error leakage, and SSE anomalies. |

AI Agent Security

AI agent security tools assess memory, tools, permissions, workflows, and runtime boundaries used by autonomous and tool-enabled AI systems.

| Tool | Description |
|------|-------------|
| Agent Memory Guard | OWASP project pro a runtime defense layer that screens AI agent memory reads and writes against prompt injection, secret leakage, and memory poisoning |
| Agent-Wiz | Python CLI by Repello AI for extracting agentic workflows from LangChain/LangGraph/CrewAI/AutoGen and running automated threat modeling |
| HOL Guard | Open-source local-first runtime security layer for AI agents that e supported tool actions and local artifacts for prompt injection, secret exposure, unsafe commands, package risks, and MCP threats |
| Prismor | Self-hosted runtime control plane that screens AI coding-agent tool calls against a signed policy before execution, blocking or routing to human appro exfiltration, prompt-injection-dri, and destructi across Claude Code, Codex, and MCP servers |
| Bifrost | Go-native AI gateway that enforces guardrails, rate limits, and fine-grained access controls across LLM and MCP traffic. |
| Humanbound | Open-source CLI that sends LLM-generated ad attacks to an agent's HTTP endpoint and scores results against OWASP LLM Top 10, OWASP Agentic Top 10, NIST, and EU AI Act mappings |

Pri Machine Learning

Pri machine learning tools help limit sensiti exposure through differential privacy, encrypted computation, anonymization, and privacy testing.

| Tool | Description |
|------|-------------|
| TensorFlow Privacy | Library of pri machine learning algorithms and tools |
| OpenDP | The core library of differential privacy algorithms powering the OpenDP Project |

AI Supply-Chain Security

AI supply-chain security tools protect model artifacts, pro, dependencies, signatures, bills of materials, and deployment pipelines.

| Tool | Description |
|------|-------------|
| Model Transparency | Generate model signing metadata for pro |
| BomLens | Local-first SBOM generator that builds CycloneDX ML-BOMs for Hugging Face models, with license and known- reports |

Model Testing, Monitoring, and E

These projects support model , monitoring, robustness, safety, and comparati, including both general-purpose and security-focused assessment workflows.

| Tool | Description |
|------|-------------|
| TensorFlow Model Analysis | A library for analyzing, , and monitoring machine learning models in production|
| CircleGuardBench| A full-fledged benchmark for e protection capabilities of AI models|

Additional Open-Source Tools

The following tools and research prototypes address additional MLSecOps testing, monitoring, defense, and assurance use cases pending more granular classification.

| Tool | Description |
|------|-------------|
| MLSploit | Cloud framework for interacti with ad machine learning research |
| Ad | Python toolbox for ad robustness research |
| MITRE ATLAS | Knowledge base of ad tactics and techniques against AI systems; successor to the deprecated Ad ML Threat Matrix |
| Cle | A library of ad examples and defenses for machine learning models|
| AdvBox | Advbox is a toolbox to generate ad examples that fool neural networks in PaddlePaddle、PyTorch、Caffe2、MxNet、Keras、TensorFlow|
| Audit AI | Bias Testing for Generalized Machine Learning Applications|
| Deep Pwning | Deep-pwning is a lightweight framework for experimenting with machine learning models with the goal of e their robustness against a motivated ad|
| Privacy Meter | An open-source library to audit data privacy in statistical and machine learning algorithms|
| PromptInject | A framework that assembles ad prompts|
| TextAttack | TextAttack is a Python framework for ad attacks, data augmentation, and model training in NLP|
| OpenAttack | An Open-Source Package for Textual Ad Attack|
| TextFooler | A Model for Natural Language Attack on Text Classification and Inference|
| Flawed Machine Learning Security | Practical examples of "Flawed Machine Learning Security" together with ML Security best practice across the end to end stages of the machine learning model lifecycle from training, to packaging, to deployment|
| Ad Machine Learning CTF | CTF challenge demonstrating ad attacks against neural-network image classifiers |
| Damn LLM Project | A Large Language Model designed for getting hacked|
| Gandalf Lakera | Prompt Injection CTF playground|
| Vigil | LLM prompt injection and security scanner|
| PALLMs (Payloads for Attacking Large Language Models) | list of payloads for attacking LLMs collected in one place|
| AI-exploits | Exploits for MLOps systems, extending beyond inputs provided to LLMs such as ChatGPT |
| Offensive ML Playbook | Offensive ML Playbook. Notes on machine learning attacks and pentesting|
| AnonLLM | Anonymize Personally Identifiable Information (PII) for Large Language Model APIs|
| AI Goat | LLM CTF challenges|
| Pyrit | The Python Risk Identification Tool for generative AI|
| Raze to the Ground: Query-Efficient Ad HTML Attacks on Machine-Learning Phishing Webpage Detectors | Source code of the paper "Raze to the Ground: Query-Efficient Ad HTML Attacks on Machine-Learning Phishing Webpage Detectors" accepted at AISec '23|
| Giskard | Open-source testing tool for LLM applications|
| Model-In | A framework for implementing Model In attacks|
| NeMo Guardrails | NeMo Guardrails allow de building LLM-based applications to add programmable guardrails between the application code and the LLM |
| AugLy | Multimodal data-augmentation library for e and improving model robustness; it is not an ad generator |
| Knockoffnets | Proof of concept for training a functional surrogate from black-box model queries |
| VGER | Jupyter Attack framework |
| AIShield Watchtower | An open-source tool from AIShield for studying AI models and scanning for |
| PS-fuzz | tool for scanning LLM |
| Mindgard CLI | E the security of AI systems through a CLI |
| Purple Llama| Meta's umbrella suite of LLM safety and security tooling, including Llama Guard, Code Shield, and CyberSecEval |
| ARTkit|Automated prompt-based testing and e of Gen AI applications|
| LangBiTe | A Bias Tester framework for LLMs|
| TF Encrypted| Framework for encrypted deep learning in TensorFlow using secure multi-party computation|
| Agentic Security| Agentic LLM Scanner / AI red teaming kit|
| skill-audit-mcp | Static security scanner for MCP servers, AI agent skills, and plugins, with SARIF output and CI integration |
| AIsbom | Disassembles Pickle bytecode and parses SafeTensors/GGUF binary headers to detect malware and license risks in ML model files before load. Generates CycloneDX/SPDX SBOMs. |
| AI-Scan-Interceptor | Self-hostable DLP gateway for enterprise prompts to ChatGPT/Claude/Gemini (Squid + Go ICAP + mTLS, AGPL-3.0) |
| NB Defense | Jupyter notebook security scanner from Protect AI (now part of Palo Alto Networks) |
| TrustGate | An open-source Generati Firewall (GAF) |
| Whistleblower | Open-source offensive tool by Repello AI for testing LLM apps against system prompt leakage |
| IronClaw | Self-hosted runtime that sandboxes autonomous AI agents with isolated execution, restricted networking, and human-appro |

<a id="commercial-tools"></a>

Commercial MLSecOps and AI Security Tools

Commercial MLSecOps and AI security tools support model protection, application testing, monitoring, go, and incident response. E technical documentation, deployment options, integrations, and independent e before adoption.

| Tool | Description |
|------|-------------|
| Citadel Lens | Commercial model-testing platform; supported standards and security checks in current product documentation |
| Cisco AI Defense (formerly Robust Intelligence) | Commercial platform for AI model and runtime protection; Robust Intelligence was acquired by Cisco |
| HiddenLayer AI Detection and Response | Commercial AI detection and response platform from HiddenLayer |
| Guardian | Model protection in CI/CD |
| NeuralTrust | Tools to protect, secure and test GenAI Applications |
| SourceryKit | an AI agent's outbound requests and MCP handoffs against a source of truth using zero-knowledge proofs, so a call only goes out if its claims check out; hooks the HTTP libraries, logs each outbound call, and blocks endpoints not on the trusted allow-list |
| Trent AI | Agentic AI security platform that continuously assesses AI agents, MCP servers, LLM and AI-nati, and code shipped with AI coding tools, traces attack chains, and proposed fixes landed |

<a id="data"></a>

Data Privacy and Anonymization Tools

Data privacy and anonymization tools help reduce exposure of personal or sensiti in machine learning datasets, images, , and model interactions.

| Tool | Description |
|------|-------------|
| ARX - Data Anonymization Tool | Tool for anonymizing datasets |
| Data-Veil | Data masking and anonymization tool |
| Tool for IMG anonymization| Image anonymization|
| Tool for DATA anonymization| Data anonymization|
| BMW Anonymization API | This repository allows you to anonymize sensiti in images/. The solution is fully compatible with the DL-based training/inference solutions that we already published/will publish for Object Detection and Semantic Segmentation |
| DeepPrivacy2| A Toolbox for Realistic Image Anonymization |

ML Code Security

ML code security resources help practitioners identify dependencies, unsafe model formats, insecure library behavior, and model-extraction risks in machine learning software.

<a id="101-resources"></a>

AI Security and MLSecOps Beginner Resources

These foundational resources explain how ad attack AI systems and how defenders secure models, prompts, data, and MLOps workflows.

AI Security Study Map

![AI Security Study Map]()

Full size map in this repository

Threat Modeling

AI threat modeling identifies assets, trust boundaries, attack paths, and security controls across models, data, prompts, pipelines, infrastructure, and agent capabilities.

!image

!image

!image

!image

!image

More in Ad AI Attacks, Mitigations, and Defense Strategies: A cybersecurity professional's guide to AI attacks, threat modeling, and securing AI with MLSecOps.

Attack

Machine learning systems can be attacked through poisoned data, ad inputs, model extraction, privacy attacks, compromised artifacts, excessi permissions, and MLOps infrastructure. The resources below explain indi attack classes and their defensi.

Blogs and Publications

These publications co practices, ad machine learning research, LLM security, AI red teaming, privacy, and security measurement.

MLOps Infrastructure

MLOps infrastructure introduces security risks across notebooks, training pipelines, model registries, artifact formats, cloud services, Kubernetes workloads, and inference endpoints. These resources document , exploits, and defensi affecting the operational ML lifecycle.

MLSecOps Pipeline

!image

<a id="repositories"></a>

Ad Machine Learning Research Repositories

These research repositories provide implementations, experiments, benchmarks, attacks, and defenses for ad machine learning, model privacy, extraction, in, robustness, and LLM security.

| Repository | Category | Security focus |
|---|---|---|
| <a id="agentpoison"></a>AgentPoison | LLM agent backdoor poisoning | Red-teaming LLM agents or knowledge-base poisoning attacks. |
| <a id="deeppayload"></a>DeepPayload | Neural trojan research | Explores embedding malicious payload behaviors into deep neural networks. |
| <a id="backdoor"></a>backdoor | Backdoor attacks | In hidden-trigger backdoor attacks in deep learning models. |
| <a id="stealing_dl_models"></a>Stealing_DL_Models | Model stealing | Demonstrates model-stealing techniques against deep learning systems. |
| <a id="datafree-model-extraction"></a>datafree-model-extraction | Model extraction | Studies data-free extraction of target model behavior. |
| <a id="llmmap"></a>LLMmap | LLM attack-surface mapping | Maps and analyzes LLM behavior to support security-oriented assessment. |
| <a id="googlecloud-federated-ml-pipeline"></a>GoogleCloud-Federated-ML-Pipeline | Federated learning pipelines | Implements a federated machine-learning pipeline using Google Cloud infrastructure. |
| <a id="class_actiattack"></a>Class_ActiAttack | Ad e attacks | Uses CAM-based ensemble methods to craft ad examples. |
| <a id="cold-attack"></a>COLD-Attack | Ad optimization attacks | Implements constrained optimization attacks against deep models. |
| <a id="pal"></a>pal | Adapti attacks | Research code for adapti strategies against defended models. |
| <a id="zeroshotknowledgetransfer"></a>ZeroShotKnowledgeTransfer | Model extraction | Zero-shot knowledge-transfer methods relevant to model-stealing risk. |
| <a id="gmi-attack"></a>GMI-Attack | Model in attacks | Implements generati in attacks against trained models. |
| <a id="knowledge-enriched-dmi"></a>Knowledge-Enriched-DMI | Model in attacks | Extends deep model in with auxiliary knowledge priors. |
| <a id="vmi"></a>vmi | Model in attacks | Implements model in attacks to reco training information. |
| <a id="plug-and-play-attacks"></a>Plug-and-Play-Attacks | Black-box attack methods | Provides plug-and-play attack pipelines for black-box ML models. |
| <a id="snap-sp23"></a>snap-sp23 | Privacy extraction attacks | Implements SNAP poisoning attacks for pri extraction from models. |
| <a id="pri"></a>pri | Privacy research | E trade-offs between differential privacy and ad robustness. |
| <a id="ml-leaks"></a>ML-Leaks | Membership inference attacks | Implements membership-inference attacks to test privacy leakage. |
| <a id="blindmi"></a>BlindMI | Membership inference attacks | Studies label-only and blind membership-inference techniques. |
| <a id="python-dp-dl"></a>python-DP-DL | Privacy defenses | Differential privacy utilities for protecting training data in deep learning. |
| <a id="mmd-mixup-defense"></a>MMD-mixup-Defense | Ad defenses | Defense approach designed to impro against ad attacks. |
| <a id="memguard"></a>MemGuard | Membership inference defenses | Defends model outputs against membership-inference attacks. |
| <a id="unsplit"></a>unsplit | Split-learning attacks | Demonstrates model in, model stealing, and label inference against split learning. |
| <a id="face_attribute_attack"></a>face_attribute_attack | Biometric ad attacks | Explores attacks on face-attribute recognition systems. |
| <a id="fvb"></a>FVB | Biometric ad attacks | Attack methods targeting face- models. |
| <a id="malware-gan"></a>Malware-GAN | Malware e research | Uses GAN-generated to study ML-based malware-detection e. |
| <a id="generatiions"></a>Generatiions | Ad example generation | Generates ad perturbations with generati. |
| <a id="adelati"></a>Adelati | Ad example generation | Implements Ad ad attacks for deep models. |
| <a id="llm-attacks"></a>llm-attacks | LLM jailbreak research | Research code for ad prompting and jailbreak attacks on LLMs. |
| <a id="llms-finetuning-safety"></a>LLMs-Finetuning-Safety | LLM safety defenses | Studies safer LLM fine-tuning under harmful-behavior and leakage risks. |
| <a id="decodingtrust"></a>DecodingTrust | Trust and safety benchmarking | Benchmark suite for e trust, safety, and robustness dimensions in LLMs. |
| <a id="promptbench"></a>promptbench | Prompt robustness benchmarking | Benchmarking framework for e prompt robustness and reliability. |
| <a id="rome"></a>rome | Model editing and reliability | Implements factual model-editing methods used in reliability and safety research. |
| <a id="llmprivacy"></a>llmprivacy | LLM privacy attacks | Research on privacy leakage and protection in large language models. |

Community Resources

MLSecOps communities and security frameworks connect practitioners with threat knowledge, standards, incident data, research, and peer collaboration.

Books

These books provide in-depth guidance on ad AI, pri machine learning, threat modeling, and generative AI security.

Infographics

MLSecOps Lifecycle

AI Security Market Map

Contributions

All contributions to this list are welcome! Please feel free to submit a pull request with any additions or impro.

Notable additions are featured in The MLSecOps Hacker newsletter, so a merged entry also reaches its subscribers.

Contributors ✨

As of July 28, 2026, this project recognizes 19 contributors:

<table>
<tr>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b><br />_{GitHub Copilot coding agent}</a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
</tr>
<tr>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt="-titan"/><br />_{<b>-titan</b>}</a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
</tr>
<tr>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt="@rob"/><br />_<b>@rob</b></a></td>
</tr>
<tr>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
<td align="center"><a href=" src=" width="100px;" alt="-ssp"/><br />_<b>-ssp</b></a></td>
<td align="center"><a href=" src=" width="100px;" alt=""/><br />_<b></b></a></td>
</tr>
</table>

Repository Stats

!GitHub stars
!GitHub forks
!GitHub watchers
!GitHub last commit
!GitHub issues
!GitHub pull requests

Activity

!Repo activity
!Contributors

Support Us

If you find this project useful, please consider giving it a star ⭐️ and subscribing to The MLSecOps Hacker 📬, the maintainer's free newsletter that funds the time spent curating this list.

![GitHub Sponsor]()

Cite This Repository

Awesome MLSecOps is a citable, machine-readable catalog. Citation metadata lives in CITATION.cff, which GitHub exposes through the Cite this repository button on the repository page (APA and BibTeX). Reference managers such as Zotero and Zenodo read the same file.

If you reference this catalog in a paper, blog post, course, or internal security guideline, please cite it as:

> Biosas, R., & Awesome MLSecOps contributors. (2023). *Awesome MLSecOps: Machine Learning and AI Security Resources* [Dataset]. GitHub.

License

This project is licensed under the MIT License - see the LICENSE file for details.

![MIT License]()

---

<p align="center">Made with ❤️ · <a href=" to The MLSecOps Hacker</a></p>

项目信息

开发语言：Astro

Star 数：473

技术要点

该项目属于红蓝攻防方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕A curated list of MLSecOps tools and resources for securing machine learning and AI systems - adversarial ML defense, LLM security, AI red teaming, model scanning, supply-chain protection, and MLOps pipeline security.展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。