Skip to content

Commit f17edff

Browse files
Marshalclaude
authored and
Marshal
committed
Initial commit: AI Security Testing Playbook
Adds full playbook with docs, checklists, attack patterns, labs, and CI/CD workflows for AI/LLM security testing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
0 parents  commit f17edff

44 files changed

Lines changed: 1146 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: basic-ci
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
jobs:
8+
sanity:
9+
runs-on: ubuntu-latest
10+
steps:
11+
- uses: actions/checkout@v4
12+
13+
- name: Check repository structure
14+
run: |
15+
test -f README.md
16+
test -f LICENSE
17+
test -f mkdocs.yml
18+
test -f docs-requirements.txt
19+
test -d playbooks
20+
test -d checklists
21+
test -d labs
22+
23+
- name: Python syntax check (labs)
24+
run: |
25+
python -m compileall labs

‎.github/workflows/pages.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
name: deploy-docs
2+
3+
on:
4+
push:
5+
branches: [ "main" ]
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
pages: write
11+
id-token: write
12+
13+
concurrency:
14+
group: "pages"
15+
cancel-in-progress: true
16+
17+
jobs:
18+
build:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- uses: actions/checkout@v4
22+
23+
- name: Set up Python
24+
uses: actions/setup-python@v5
25+
with:
26+
python-version: "3.x"
27+
28+
- name: Install MkDocs
29+
run: |
30+
python -m pip install --upgrade pip
31+
pip install -r docs-requirements.txt
32+
33+
- name: Build site
34+
run: mkdocs build --strict
35+
36+
- name: Upload artifact
37+
uses: actions/upload-pages-artifact@v3
38+
with:
39+
path: site
40+
41+
deploy:
42+
environment:
43+
name: github-pages
44+
url: ${{ steps.deployment.outputs.page_url }}
45+
runs-on: ubuntu-latest
46+
needs: build
47+
steps:
48+
- name: Deploy to GitHub Pages
49+
id: deployment
50+
uses: actions/deploy-pages@v4

‎.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
__pycache__/
2+
*.pyc
3+
.venv/
4+
venv/
5+
.env
6+
.DS_Store
7+
.idea/
8+
.vscode/
9+
site/

‎CODE_OF_CONDUCT.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Code of Conduct
2+
3+
Be respectful, inclusive, and constructive.
4+
Harassment or abusive behavior is not tolerated.
5+
Assume good intent and help improve the quality and safety of the content.

‎CONTRIBUTING.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Contributing
2+
3+
## What to contribute
4+
- New defensive test cases
5+
- Mitigation patterns
6+
- Checklists and review guides
7+
- Labs that run locally with synthetic data
8+
9+
## How to contribute
10+
1) Fork + branch
11+
2) Keep content defensive and authorized-use focused
12+
3) Submit PR with a short description
13+
14+
## Style
15+
- Use clear headings
16+
- Include "Expected outcome" in test cases
17+
- Avoid real-world exploit instructions

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
![](assets/banner.svg)
2+
3+
# AI Security Testing Playbook
4+
5+
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
6+
[![Contributions Welcome](https://img.shields.io/badge/Contributions-Welcome-blue.svg)](CONTRIBUTING.md)
7+
[![Security Policy](https://img.shields.io/badge/Security-Policy-orange.svg)](SECURITY.md)
8+
9+
A practical, defensive playbook for **testing and securing LLM-powered apps** (chatbots, RAG systems, agentic tools, code assistants) in **authorized environments**.
10+
11+
This repository focuses on:
12+
- **How to test** AI systems for common security failures
13+
- **What to log + measure**
14+
- **How to mitigate** issues with practical patterns
15+
- **Hands-on labs** you can run locally
16+
17+
> ⚠️ Ethics & Scope: This repo is for security testing on systems you own or have explicit permission to test. See **[docs/scope-and-ethics.md](docs/scope-and-ethics.md)**.
18+
19+
---
20+
21+
## Quick links
22+
23+
- **Playbooks** → [playbooks/](playbooks/)
24+
- **Checklists** → [checklists/](checklists/)
25+
- **Mitigation patterns** → [patterns/mitigation-patterns.md](patterns/mitigation-patterns.md)
26+
- **Local lab** → [labs/prompt-injection-toy-app/](labs/prompt-injection-toy-app/)
27+
- **Threat modeling** → [docs/threat-modeling.md](docs/threat-modeling.md)
28+
29+
---
30+
31+
## The Top 10 LLM App Security Risks (practical)
32+
33+
1. **Prompt injection** (direct + indirect via docs)
34+
2. **Tool abuse** (unsafe actions, privilege misuse)
35+
3. **Tool-output injection** (model trusts tool output as instructions)
36+
4. **RAG overexposure** (retrieves sensitive docs / too-broad scope)
37+
5. **RAG poisoning** (malicious documents / source spoofing)
38+
6. **Sensitive data leakage** (system prompts, memory, logs)
39+
7. **Authz gaps** (model can access data the user shouldn’t)
40+
8. **Insecure AI-generated code** (weak crypto, injection, auth flaws)
41+
9. **Unsafe defaults in production** (no rate limits, no monitoring)
42+
10. **Evaluation blind spots** (no regression tests for security failures)
43+
44+
Use the checklists here to systematically test each category.
45+
46+
---
47+
48+
## Reference architecture (where attacks happen)
49+
50+
```text
51+
Untrusted Inputs
52+
(user, files, URLs, tool outputs)
53+
|
54+
v
55+
+-------------+
56+
| LLM APP | <-- prompt assembly, policy, routing
57+
+-------------+
58+
| | |
59+
| | +--> RAG (retrieval + docs)
60+
| +--------> Tools (APIs / actions)
61+
+--------------> Response (user)
62+
```
63+
64+
Key idea: **treat anything untrusted as data**, and strictly control how it reaches prompts and tools.
65+
66+
---
67+
68+
## What’s inside
69+
70+
### Playbooks
71+
- Prompt Injection: [playbooks/prompt-injection.md](playbooks/prompt-injection.md)
72+
- Jailbreaks: [playbooks/jailbreaks.md](playbooks/jailbreaks.md)
73+
- Data Leakage: [playbooks/data-leakage.md](playbooks/data-leakage.md)
74+
- Tool / Agent Security: [playbooks/tool-use-security.md](playbooks/tool-use-security.md)
75+
- RAG Security: [playbooks/rag-security.md](playbooks/rag-security.md)
76+
- Code Generation Security: [playbooks/code-generation-security.md](playbooks/code-generation-security.md)
77+
- Incident Response: [playbooks/incident-response.md](playbooks/incident-response.md)
78+
79+
### Checklists
80+
- AI Red Teaming Checklist: [checklists/ai-red-teaming-checklist.md](checklists/ai-red-teaming-checklist.md)
81+
- LLM App Security Review: [checklists/llm-app-security-review.md](checklists/llm-app-security-review.md)
82+
- Secure Prompting Review: [checklists/secure-prompting-review.md](checklists/secure-prompting-review.md)
83+
84+
### Patterns & Metrics
85+
- Attack Taxonomy: [patterns/attack-taxonomy.md](patterns/attack-taxonomy.md)
86+
- Mitigation Patterns: [patterns/mitigation-patterns.md](patterns/mitigation-patterns.md)
87+
- Eval Metrics: [patterns/eval-metrics.md](patterns/eval-metrics.md)
88+
- Logging & Monitoring: [patterns/logging-and-monitoring.md](patterns/logging-and-monitoring.md)
89+
90+
### Labs (local)
91+
- Prompt Injection Toy App: [labs/prompt-injection-toy-app/README.md](labs/prompt-injection-toy-app/README.md)
92+
- RAG Poisoning Simulator: [labs/rag-poisoning-simulator/README.md](labs/rag-poisoning-simulator/README.md)
93+
- Tool Output Injection Simulator: [labs/tool-output-injection-simulator/README.md](labs/tool-output-injection-simulator/README.md)
94+
95+
---
96+
97+
## Quickstart
98+
99+
1) Read the guardrails:
100+
- [docs/scope-and-ethics.md](docs/scope-and-ethics.md)
101+
102+
2) Run a lab locally:
103+
- [labs/prompt-injection-toy-app/README.md](labs/prompt-injection-toy-app/README.md)
104+
105+
3) Use a checklist during reviews:
106+
- [checklists/llm-app-security-review.md](checklists/llm-app-security-review.md)
107+
108+
---
109+
110+
## Optional: GitHub Pages docs (MkDocs)
111+
This repo includes an **MkDocs** config so you can publish docs via GitHub Pages easily:
112+
- `mkdocs.yml`
113+
- `docs/index.md`
114+
115+
To build locally:
116+
```bash
117+
pip install -r docs-requirements.txt
118+
mkdocs serve
119+
```
120+
121+
---
122+
123+
## Contributing
124+
PRs welcome. Please read:
125+
- [CONTRIBUTING.md](CONTRIBUTING.md)
126+
- [SECURITY.md](SECURITY.md)
127+
128+
---
129+
130+
## License
131+
MIT — see [LICENSE](LICENSE).
132+
133+
134+
## Prompt Injection Attack Examples
135+
See: examples/prompt-injection-attacks.md

‎SECURITY.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Security Policy
2+
3+
If you believe this repository contains content that could be misused or is too operational for wrongdoing:
4+
- Open an issue tagged "security"
5+
- Or propose a PR to reframe it as defensive-only
6+
7+
We aim to keep this repository focused on authorized testing and mitigations.

‎assets/banner.svg‎

Lines changed: 10 additions & 0 deletions
Loading

‎assets/logo.svg‎

Lines changed: 8 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)