False positives for repository documentation read by default agent workflows

[orassayag]

Hi,

First of all, thanks for building Agent Scan - it's been very useful.

I noticed what appears to be a class of false positives when scanning default agent workflows (e.g. Claude Code/Codex skills).

For example, Agent Scan reports:

[W011 high]: Third-party content exposure detected (high risk: 0.85).

The required workflow explicitly instructs the agent to "Explore the codebase" and "inventory existing documentation" (e.g. docs/**/*.md, CONTRIBUTING.md, ARCHITECTURE.md, README.md), which are outsider-authored free text from the repository, and that text would be read at runtime to generate/update the customization files.

The issue is that these instructions are part of the normal behavior of coding agents. Reading repository files such as README.md, CONTRIBUTING.md, ARCHITECTURE.md, or documentation under docs/ is generally expected and necessary for understanding the project before making changes.

More (Codex - Installed with the CLI by default):

├── skill-installer 1 high finding
│   ● [W013 high]: Attempt to modify system services in skill instructions detected (high risk: 0.90). The prompt explicitly tells the agent to "request escalation" when running the helper scripts in the sandbox (encouraging privilege escalation) and includes 
│   instructions that allow overwriting preinstalled system skills and accessing private repos, which can modify the machine state.
│   ├── instruction SKILL.md                             
│   ├── asset       openai.yaml                          
│   ├── asset       LICENSE.txt                          
│   ├── asset       skill-installer.png                  
│   ├── asset       skill-installer-small.svg            
│   ├── script      github_utils.py                      
│   ├── script      list-skills.py                       
│   └── script      install-skill-from-github.py         
├── openai-docs 4 findings (4 high)
│   ● [E004 high]: Potential prompt injection detected (high risk: 1.00). The skill includes an instruction to retry an install "with escalated permissions" and "Do not ask the user to run it yet," which directs the agent to attempt privilege escalation and 
│   bypass user consent—a deceptive action outside the docs-only guidance scope.
│   ● [E006 high]: Malicious code pattern detected (high risk: 1.00). The skill contains an explicit instruction to automatically retry an installation with escalated permissions and to avoid asking the user to run it, which is an intentional 
│   privilege-escalation and social-engineering pattern indicating high-risk/malicious intent.
│   ● [W012 high]: Potentially malicious external URL detected (high risk: 1.00). The skill and its helper script explicitly fetch and use https://developers.openai.com/api/docs/guides/latest-model.md (and the MCP endpoint https://developers.openai.com/mcp) at 
│   runtime to determine model IDs and to fetch migration/prompting guides that directly control model-selection and prompt-upgrade behavior.
│   ● [W013 high]: Attempt to modify system services in skill instructions detected (high risk: 1.00). The skill explicitly instructs retrying an install command "with escalated permissions" if it fails due to permissions/sandboxing, which directs the agent to 
│   obtain elevated privileges and bypass security controls.

As a result, scans of the default skills/prompts shipped with tools like Claude Code or Codex generate high-severity findings even though there doesn't appear to be an actual prompt injection or data exposure issue.

Is this expected behavior?
I know i can just scan specific paths but I think it might worth (if so):

• lowering the severity for this pattern,
• distinguishing expected repository-documentation reads from genuine third-party prompt ingestion,
• or providing a suppression/allowlist for common documentation files.

Otherwise, it may produce a significant number of false positives that make it harder to identify real issues.

Thanks!

[jiezeng2004-design]

A useful distinction here may be mention, instruction, and reachable execution rather than treating every security-related phrase in repository documentation as equivalent:

1. Security documentation often mentions strings such as rm -rf, credential paths, exfiltration commands, or privilege escalation specifically to explain that they must be blocked.
2. Normal coding-agent instructions ask the agent to read repository-owned README.md, CONTRIBUTING.md, or docs/** before making changes.
3. Higher confidence should require a reachable instruction or executable path that combines untrusted content with a sensitive action, external transfer, or consent bypass.

Package inclusion and runtime reachability seem useful classification signals. A malicious-looking fixture outside the shipped package, a quoted negative example in a threat model, and an instruction executed by an installed skill should not necessarily receive the same confidence or severity. User consent and whether the content can actually reach a tool call are also materially different signals.

PatchWarden is a useful non-executing example for this class of test because its security documentation intentionally contains destructive commands, sensitive paths, prompt-injection language, and exfiltration examples while the product's purpose is to block or approve those operations. A static fixture should use synthetic metadata and documentation only; Agent Scan should not start the patchwarden stdio MCP process merely to classify the documentation.

Would an anonymized JSON result plus a minimal synthetic documentation fixture be useful for calibrating W011/W013/E004/E006? I am asking before preparing data because the repository's current contribution policy says external code contributions are closed.

[orassayag]

@jiezeng2004-design

This is a really clear way to frame it — thanks for putting it into words. The "mention vs. instruction vs. reachable execution" split maps exactly onto what I was running into: Agent Scan currently seems to weight presence of security-relevant strings heavily, without checking whether that content is actually wired into something that gets executed or acted on by the agent.

Your PatchWarden example is a good illustration too — a tool whose whole job is to document/handle dangerous operations will always look "high risk" under a mentions-based heuristic, even though the actual risk depends on package inclusion and reachability, exactly as you said.

On your question: yes, I think an anonymized JSON scan result plus a minimal synthetic documentation fixture would be genuinely useful for calibrating W011/W013/E004/E006 — it would give the maintainers something concrete to test severity/confidence adjustments against, without needing real repo content.

That said, since you flagged that the contribution policy currently has external code contributions closed, it's probably worth a maintainer confirming whether that also covers test fixtures/data submitted for calibration purposes (as opposed to a PR against the detection logic itself) — cc'ing in case someone from the Agent Scan team wants to weigh in before you spend time preparing it.

[pdugan20]

Adding three concrete W011 cases from public skills.sh scans of pdugan20/skills:

• bootstrap-repository: medium W011 because the skill reads GitHub repository metadata and rulesets. Those values are required inspection inputs; removing the read would remove the core capability.
• review-mintlify-docs: medium W011 because the skill reads hand-authored MDX and docs.json. That content is the material being reviewed.
• feature-spike: medium W011 because the runtime default prompt receives the user feature description. Passing the user request into the selected skill appears to be expected operation for essentially every agent skill.

Pages:

• https://www.skills.sh/pdugan20/skills/bootstrap-repository
• https://www.skills.sh/pdugan20/skills/review-mintlify-docs
• https://www.skills.sh/pdugan20/skills/feature-spike

All three currently pass the Agent Trust Hub and Socket checks. We are adding narrow runtime trust boundaries and adversarial evals: ordinary repository content, API responses, documentation, and generated output remain usable data, but cannot independently authorize commands, secret access, external writes, or scope expansion. Legitimate user requests and applicable agent instruction files remain authoritative.

These examples support the distinction already proposed here between content exposure and reachable authority or execution. Can W011 account for explicit data-versus-instruction boundaries, independently validated commands, and existing approval gates, either by lowering the classification or by reporting the exposure as informational? In particular, the feature-spike result seems broader than the repository-documentation case because it treats receipt of the initiating user prompt itself as third-party exposure.