MLSecOps (Machine Learning Security Operations) is the practice of integrating security throughout the machine learning lifecycle—from data collection and model development to deployment, monitoring, and incident response. It applies security testing, threat modeling, supply-chain protection, access controls, and continuous monitoring to machine learning models, MLOps pipelines, LLM applications, and AI agents.
This curated catalog helps security engineers, ML practitioners, developers, and AI red teams discover open-source MLSecOps tools, adversarial machine learning research, AI security frameworks, threat-modeling resources, and practical learning materials.
⭐ If this catalog is useful, star the repository or read the contribution guidelines to suggest a resource.
- What is MLSecOps?
- Open Source Security Tools
- Commercial MLSecOps and AI Security Tools
- Data Privacy and Anonymization Tools
- ML Code Security
- AI Security and MLSecOps Beginner Resources
- Threat Modeling
- Attack Vectors
- Blogs and Publications
- MLOps Infrastructure Vulnerabilities
- MLSecOps Pipeline
- Adversarial Machine Learning Research Repositories
- Community Resources
- Books
- Infographics
- Contributions
- Contributors
- Repository Stats
- Activity
- Support Us
- License
Open-source MLSecOps tools help practitioners test, monitor, and protect machine learning models, data, pipelines, LLM applications, and AI agents. The resources below include active projects and proofs of concept; evaluate maintenance status, licensing, threat coverage, and deployment suitability before production use.
Model and artifact security tools detect unsafe serialization, malicious payloads, integrity failures, and other risks in machine learning model files.
| Tool | Description |
|---|---|
| ModelScan | Protection Against ML Model Serialization Attacks |
| Safetensors | Tensor storage format designed to avoid executable deserialization; it does not establish model provenance or behavioral safety |
Adversarial machine learning testing tools evaluate model robustness against evasion, poisoning, extraction, inversion, and other adversarial techniques.
| Tool | Description |
|---|---|
| Adversarial Robustness Toolbox | Library of defense methods for ML models against adversarial attacks |
| Foolbox | Python toolbox for creating and evaluating adversarial attacks and defenses |
LLM security and red-teaming tools test prompts, model behavior, application controls, and safeguards against attacks such as prompt injection and jailbreaks.
| Tool | Description |
|---|---|
| Garak | LLM vulnerability scanner |
| Promptfoo Scanner | An open-source LLM red teaming tool |
AI agent security tools assess memory, tools, permissions, workflows, and runtime boundaries used by autonomous and tool-enabled AI systems.
| Tool | Description |
|---|---|
| Agent Memory Guard | OWASP project providing a runtime defense layer that screens AI agent memory reads and writes against prompt injection, secret leakage, and memory poisoning |
| Agent-Wiz | Python CLI by Repello AI for extracting agentic workflows from LangChain/LangGraph/CrewAI/AutoGen and running automated threat modeling |
Privacy-preserving machine learning tools help limit sensitive-data exposure through differential privacy, encrypted computation, anonymization, and privacy testing.
| Tool | Description |
|---|---|
| TensorFlow Privacy | Library of privacy-preserving machine learning algorithms and tools |
| OpenDP | The core library of differential privacy algorithms powering the OpenDP Project |
AI supply-chain security tools protect model artifacts, provenance, dependencies, signatures, bills of materials, and deployment pipelines.
| Tool | Description |
|---|---|
| Model Transparency | Generate model signing metadata for provenance verification |
| BomLens | Local-first SBOM generator that builds CycloneDX ML-BOMs for Hugging Face models, with license and known-vulnerability reports |
These projects support model validation, monitoring, robustness, safety, and comparative evaluation, including both general-purpose and security-focused assessment workflows.
| Tool | Description |
|---|---|
| TensorFlow Model Analysis | A library for analyzing, validating, and monitoring machine learning models in production |
| CircleGuardBench | A full-fledged benchmark for evaluating protection capabilities of AI models |
The following tools and research prototypes address additional MLSecOps testing, monitoring, defense, and assurance use cases pending more granular classification.
| Tool | Description |
|---|---|
| MLSploit | Cloud framework for interactive experimentation with adversarial machine learning research |
| Advertorch | Python toolbox for adversarial robustness research |
| MITRE ATLAS | Knowledge base of adversary tactics and techniques against AI systems; successor to the deprecated Adversarial ML Threat Matrix |
| CleverHans | A library of adversarial examples and defenses for machine learning models |
| AdvBox | Advbox is a toolbox to generate adversarial examples that fool neural networks in PaddlePaddle、PyTorch、Caffe2、MxNet、Keras、TensorFlow |
| Audit AI | Bias Testing for Generalized Machine Learning Applications |
| Deep Pwning | Deep-pwning is a lightweight framework for experimenting with machine learning models with the goal of evaluating their robustness against a motivated adversary |
| Privacy Meter | An open-source library to audit data privacy in statistical and machine learning algorithms |
| PromptInject | A framework that assembles adversarial prompts |
| TextAttack | TextAttack is a Python framework for adversarial attacks, data augmentation, and model training in NLP |
| OpenAttack | An Open-Source Package for Textual Adversarial Attack |
| TextFooler | A Model for Natural Language Attack on Text Classification and Inference |
| Flawed Machine Learning Security | Practical examples of "Flawed Machine Learning Security" together with ML Security best practice across the end to end stages of the machine learning model lifecycle from training, to packaging, to deployment |
| Adversarial Machine Learning CTF | CTF challenge demonstrating adversarial-image attacks against neural-network image classifiers |
| Damn Vulnerable LLM Project | A Large Language Model designed for getting hacked |
| Gandalf Lakera | Prompt Injection CTF playground |
| Vigil | LLM prompt injection and security scanner |
| PALLMs (Payloads for Attacking Large Language Models) | list of various payloads for attacking LLMs collected in one place |
| AI-exploits | Exploits for MLOps systems, extending beyond inputs provided to LLMs such as ChatGPT |
| Offensive ML Playbook | Offensive ML Playbook. Notes on machine learning attacks and pentesting |
| AnonLLM | Anonymize Personally Identifiable Information (PII) for Large Language Model APIs |
| AI Goat | vulnerable LLM CTF challenges |
| Pyrit | The Python Risk Identification Tool for generative AI |
| Raze to the Ground: Query-Efficient Adversarial HTML Attacks on Machine-Learning Phishing Webpage Detectors | Source code of the paper "Raze to the Ground: Query-Efficient Adversarial HTML Attacks on Machine-Learning Phishing Webpage Detectors" accepted at AISec '23 |
| Giskard | Open-source testing tool for LLM applications |
| Model-Inversion-Attack-ToolBox | A framework for implementing Model Inversion attacks |
| NeMo Guardrails | NeMo Guardrails allow developers building LLM-based applications to add programmable guardrails between the application code and the LLM |
| AugLy | Multimodal data-augmentation library for evaluating and improving model robustness; it is not an adversarial-attack generator |
| Knockoffnets | Proof of concept for training a functional surrogate from black-box model queries |
| VGER | Jupyter Attack framework |
| AIShield Watchtower | An open-source tool from AIShield for studying AI models and scanning for vulnerabilities |
| PS-fuzz | tool for scanning LLM vulnerabilities |
| Mindgard CLI | Evaluate the security of AI systems through a CLI |
| Purple Llama | Meta's umbrella suite of LLM safety and security tooling, including Llama Guard, Code Shield, and CyberSecEval |
| ARTkit | Automated prompt-based testing and evaluation of Gen AI applications |
| LangBiTe | A Bias Tester framework for LLMs |
| TF Encrypted | Framework for encrypted deep learning in TensorFlow using secure multi-party computation |
| Agentic Security | Agentic LLM Vulnerability Scanner / AI red teaming kit |
| skill-audit-mcp | Static security scanner for MCP servers, AI agent skills, and plugins, with SARIF output and CI integration |
| AIsbom | Disassembles Pickle bytecode and parses SafeTensors/GGUF binary headers to detect malware and license risks in ML model files before load. Generates CycloneDX/SPDX SBOMs. |
| AI-Scan-Interceptor | Self-hostable DLP gateway for enterprise prompts to ChatGPT/Claude/Gemini (Squid + Go ICAP + mTLS, AGPL-3.0) |
| NB Defense | Jupyter notebook security scanner from Protect AI (now part of Palo Alto Networks) |
| TrustGate | An open-source Generative Application Firewall (GAF) |
| Whistleblower | Open-source offensive tool by Repello AI for testing LLM apps against system prompt leakage |
| IronClaw | Self-hosted runtime that sandboxes autonomous AI agents with isolated execution, restricted networking, and human-approval gating |
Commercial MLSecOps and AI security tools support model protection, application testing, monitoring, governance, and incident response. Evaluate technical documentation, deployment options, integrations, and independent evidence before adoption.
| Tool | Description |
|---|---|
| Citadel Lens | Commercial model-testing platform; verify supported standards and security checks in current product documentation |
| Cisco AI Defense (formerly Robust Intelligence) | Commercial platform for AI model validation and runtime protection; Robust Intelligence was acquired by Cisco |
| HiddenLayer AI Detection and Response | Commercial AI detection and response platform from HiddenLayer |
| Guardian | Model protection in CI/CD |
| NeuralTrust | Tools to protect, secure and test GenAI Applications |
Data privacy and anonymization tools help reduce exposure of personal or sensitive information in machine learning datasets, images, videos, and model interactions.
| Tool | Description |
|---|---|
| ARX - Data Anonymization Tool | Tool for anonymizing datasets |
| Data-Veil | Data masking and anonymization tool |
| Tool for IMG anonymization | Image anonymization |
| Tool for DATA anonymization | Data anonymization |
| BMW Anonymization API | This repository allows you to anonymize sensitive information in images/videos. The solution is fully compatible with the DL-based training/inference solutions that we already published/will publish for Object Detection and Semantic Segmentation |
| DeepPrivacy2 | A Toolbox for Realistic Image Anonymization |
ML code security resources help practitioners identify vulnerable dependencies, unsafe model formats, insecure library behavior, and model-extraction risks in machine learning software.
- lintML - Security linter for ML training code, maintained by a member of the NVIDIA AI red team
- HiddenLayer: Model as Code - Research article on executable model formats and unsafe deserialization paths in ML libraries
- Copycat CNN - Proof of concept for training a surrogate convolutional neural network from black-box queries
- differential-privacy-library - Library designed for differential privacy and machine learning
These foundational resources explain how adversaries attack AI systems and how defenders secure models, prompts, data, and MLOps workflows.
- AI Security 101
- Web LLM attacks
- Microsoft AI Red Team
- AI Risk Assessment for ML Engineers
- Microsoft - Generative AI Security for Beginners
Full size map in this repository
AI threat modeling identifies assets, trust boundaries, attack paths, and security controls across models, data, prompts, pipelines, infrastructure, and agent capabilities.
More in Adversarial AI Attacks, Mitigations, and Defense Strategies: A cybersecurity professional's guide to AI attacks, threat modeling, and securing AI with MLSecOps.
Machine learning systems can be attacked through poisoned data, adversarial inputs, model extraction, privacy attacks, compromised artifacts, excessive agent permissions, and vulnerable MLOps infrastructure. The resources below explain individual attack classes and their defensive implications.
- Data Poisoning
- Adversarial Prompt Exploits
- Model Inversion Attacks
- Model Evasion Attacks
- Membership Inference Exploits
- Model Stealing Attacks
- ML Supply Chain Attacks
- Model Denial Of Service
- Gradient Leakage Attacks
- Cloud Infrastructure Attacks
- Architecture and Access-control attacks
These publications cover MLSecOps practices, adversarial machine learning research, LLM security, AI red teaming, privacy, and security measurement.
- 📚 What is MLSecOps
- 🛡️ Red-Teaming Large Language Models
- 🔍 Google's AI red-team
- 🔒 The MLSecOps Top 10 vulnerabilities
- 🏴☠️ Token Smuggling Jailbreak via Adversarial Prompt
- ☣️ Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks
- 📊 We need a new way to measure AI security
- 🕵️ PrivacyRaven: Implementing a proof of concept for model inversion
- 🧠 Adversarial Prompts Engineering
- 🔫 TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
- 📋 Trail Of Bits' audit of Hugging Face's safetensors library
- 🔝 OWASP Top 10 for Large Language Model Applications
- 🔐 LLM Security
- 🔑 Is your MLOps infrastructure leaking secrets?
- 🚩 Embrace The Red, blog where show how u can hack LLM's.
- 🎙️ Audio-jacking: Using generative AI to distort live audio transactions
- 🌐 HADESS - Web LLM Attacks
- 🧰 WTF-blog - MLSecOps frameworks ... Which ones are available and what is the difference?
- 📚 DreadNode Paper Stack
- 🛡️ CircleGuardBench: New Standard for Evaluating AI Moderation Models
MLOps infrastructure introduces security risks across notebooks, training pipelines, model registries, artifact formats, cloud services, Kubernetes workloads, and inference endpoints. These resources document vulnerabilities, exploits, and defensive controls affecting the operational ML lifecycle.
- SILENT SABOTAGE - Study on bot compromise for converting Pickle to SafeTensors
- NOT SO CLEAR: HOW MLOPS SOLUTIONS CAN MUDDY THE WATERS OF YOUR SUPPLY CHAIN - Study on vulnerabilities for the ClearML platform
- Uncovering Azure's Silent Threats: A Journey into Cloud Vulnerabilities - Study on security issues of Azure MLAAS
- The MLOps Security Landscape
- Confused Learning: Supply Chain Attacks through Machine Learning Models
These research repositories provide implementations, experiments, benchmarks, attacks, and defenses for adversarial machine learning, model privacy, extraction, inversion, robustness, and LLM security.
| Repository | Category | Security focus |
|---|---|---|
| AgentPoison | LLM agent backdoor poisoning | Red-teaming LLM agents via memory or knowledge-base poisoning attacks. |
| DeepPayload | Neural trojan research | Explores embedding malicious payload behaviors into deep neural networks. |
| backdoor | Backdoor attacks | Investigates hidden-trigger backdoor attacks in deep learning models. |
| Stealing_DL_Models | Model stealing | Demonstrates model-stealing techniques against deep learning systems. |
| datafree-model-extraction | Model extraction | Studies data-free extraction of target model behavior. |
| LLMmap | LLM attack-surface mapping | Maps and analyzes LLM behavior to support security-oriented assessment. |
| GoogleCloud-Federated-ML-Pipeline | Federated learning pipelines | Implements a federated machine-learning pipeline using Google Cloud infrastructure. |
| Class_Activation_Mapping_Ensemble_Attack | Adversarial evasion attacks | Uses CAM-based ensemble methods to craft adversarial examples. |
| COLD-Attack | Adversarial optimization attacks | Implements constrained optimization attacks against deep models. |
| pal | Adaptive adversarial attacks | Research code for adaptive attack strategies against defended models. |
| ZeroShotKnowledgeTransfer | Model extraction | Zero-shot knowledge-transfer methods relevant to model-stealing risk. |
| GMI-Attack | Model inversion attacks | Implements generative model inversion attacks against trained models. |
| Knowledge-Enriched-DMI | Model inversion attacks | Extends deep model inversion with auxiliary knowledge priors. |
| vmi | Model inversion attacks | Implements variational model inversion attacks to recover private training information. |
| Plug-and-Play-Attacks | Black-box attack methods | Provides plug-and-play attack pipelines for black-box ML models. |
| snap-sp23 | Privacy extraction attacks | Implements SNAP poisoning attacks for private-property extraction from models. |
| privacy-vs-robustness | Privacy vs. robustness research | Evaluates trade-offs between differential privacy and adversarial robustness. |
| ML-Leaks | Membership inference attacks | Implements membership-inference attacks to test privacy leakage. |
| BlindMI | Membership inference attacks | Studies label-only and blind membership-inference techniques. |
| python-DP-DL | Privacy defenses | Differential privacy utilities for protecting training data in deep learning. |
| MMD-mixup-Defense | Adversarial defenses | Defense approach designed to improve robustness against adversarial attacks. |
| MemGuard | Membership inference defenses | Defends model outputs against membership-inference attacks. |
| unsplit | Split-learning attacks | Demonstrates model inversion, model stealing, and label inference against split learning. |
| face_attribute_attack | Biometric adversarial attacks | Explores attacks on face-attribute recognition systems. |
| FVB | Biometric adversarial attacks | Attack methods targeting face-verification models. |
| Malware-GAN | Malware evasion research | Uses GAN-generated variants to study ML-based malware-detection evasion. |
| Generative_Adversarial_Perturbations | Adversarial example generation | Generates adversarial perturbations with generative models. |
| Adversarial-Attacks-with-Relativistic-AdvGAN | Adversarial example generation | Implements AdvGAN-based adversarial attacks for deep models. |
| llm-attacks | LLM jailbreak research | Research code for adversarial prompting and jailbreak attacks on LLMs. |
| LLMs-Finetuning-Safety | LLM safety defenses | Studies safer LLM fine-tuning under harmful-behavior and leakage risks. |
| DecodingTrust | Trust and safety benchmarking | Benchmark suite for evaluating trust, safety, and robustness dimensions in LLMs. |
| promptbench | Prompt robustness benchmarking | Benchmarking framework for evaluating prompt robustness and reliability. |
| rome | Model editing and reliability | Implements factual model-editing methods used in reliability and safety research. |
| llmprivacy | LLM privacy attacks | Research on privacy leakage and protection in large language models. |
MLSecOps communities and security frameworks connect practitioners with threat knowledge, standards, incident data, research, and peer collaboration.
- MLSecOps
- MLSecOps Podcast
- MITRE ATLAS™ and SLACK COMMUNITY
- Slack community
- MITRE ATLAS™ (Adversarial Threat Landscape for Artificial-Intelligence Systems)
- OWASP AI Exchange
- OWASP Machine Learning Security Top Ten
- OWASP Top 10 for Large Language Model Applications
- OWASP LLMSVS
- OWASP Periodic Table of AI Security
- OWASP SLACK
- Awesome LLM Security
- Hackstery
- PWNAI
- AiSec_X_Feed
- HUNTR Discord community
- AIRSK
- AI Vulnerability Database
- Incident AI Database
- Defcon AI Village CTF
- Awesome AI Security
- MLSecOps Reference Repository
- Awesome LVLM Attack
- Awesome MLLM Safety
These books provide in-depth guidance on adversarial AI, privacy-preserving machine learning, threat modeling, and generative AI security.
- Adversarial AI Attacks, Mitigations, and Defense Strategies: A cybersecurity professional's guide to AI attacks, threat modeling, and securing AI with MLSecOps
- Privacy-Preserving Machine Learning
- Generative AI Security: Theories and Practices (Future of Business and Finance)
All contributions to this list are welcome! Please feel free to submit a pull request with any additions or improvements.
 @riccardobiosas |
 @badarahmed |
 @deadbits |
 @wearetyomsmnv |
 @anmorgan24 |
 @mik0w |
 @alexcombessie |
 @Igralino |
 @typpo |
 @robvanderveer |
If you find this project useful, please consider giving it a star ⭐️
This project is licensed under the MIT License - see the LICENSE file for details.
Made with ❤️