ci(analyzer): adopt liveness primer for all pull requests (#802)

Author: duriantaco

.github/workflows/liveness-primer.yml

@@ -3,11 +3,6 @@ name: Analyzer Blast Radius
 on:
   pull_request:
     types: [opened, synchronize, reopened, ready_for_review]
-    paths:
-      - "skylos/**"
-      - "pyproject.toml"
-      - "MANIFEST.in"
-      - ".github/workflows/liveness-primer.yml"
 
 permissions:
   contents: read
@@ -19,7 +14,6 @@ concurrency:
 jobs:
   blast-radius:
     name: Python dead-code blast radius
-    if: github.event.pull_request.draft == false
     runs-on: ubuntu-24.04
     timeout-minutes: 45
 
@@ -43,8 +37,6 @@ jobs:
         env:
           SKYLOS_REPOSITORY: https://github.com/duriantaco/skylos
           BASE_SHA: ${{ github.event.pull_request.base.sha }}
-          # The base repository owns this synthetic merge commit, including
-          # for fork PRs, so the comparison tests what would actually land.
           MERGE_SHA: ${{ github.sha }}
           REPORT_JSON: liveness-primer-report.json
           REPORT_MARKDOWN: liveness-primer-report.md

README.md

@@ -490,6 +490,20 @@ Frozen `golden-v0.2` highlights:
 For methodology, commands, competitor rows, and caveats, see
 [BENCHMARK.md](./BENCHMARK.md).
 
+### Real-project regression testing
+
+[liveness_primer](https://github.com/mcdigman/liveness_primer), created and
+maintained by [Matthew Digman](https://github.com/mcdigman), is Skylos's official
+real-project regression testing tool. On every PR, it compares the base and
+proposed merge result against the same pinned Python projects and reports
+which findings were added, removed, or changed.
+
+Read the **Analyzer Blast Radius** check for the comparison and downloadable
+reports. These results complement the labeled benchmarks above; a change in
+finding counts alone does not establish accuracy. See the
+[liveness_primer guide](./docs/liveness-primer.md) for scope, review steps, and
+reproduction commands.
+
 ## Project Evidence
 
 Skylos-assisted dead-code cleanup PRs have been merged in

corpus/README.md

@@ -14,11 +14,11 @@ Acceptance rules:
 - Avoid whole-project assertions and avoid gating on total finding counts.
 - Prefer cases that protect common frameworks, must-not-miss hooks, and critical static-analysis edge cases.
 
-Why this is better than scanning whole upstream repositories on every PR:
+How this complements [liveness_primer](../docs/liveness-primer.md):
 
-- Whole-repo scans are noisy because mature libraries may contain legitimate findings unrelated to the regression you are trying to catch.
-- Whole-repo scans are slow and brittle because upstream code changes over time.
-- Distilled fixtures let us assert exact expectations like "this symbol must not be reported as dead code" without hiding unrelated issues.
+- The primer compares two Skylos revisions on the same pinned upstream projects. It shows real-project changes without requiring every existing finding to be labeled.
+- Corpus Guard checks small, explicit expectations like "this symbol must not be reported as dead code." Those expectations let CI distinguish a regression from an intended change.
+- When primer review uncovers a bug, reduce it to a regression test or corpus fixture here. Keep both checks: broad change detection and precise correctness checks.
 
 How to add a case:
 

docs/README.md

@@ -4,6 +4,7 @@ This repository keeps lightweight local docs for features that need examples clo
 
 | Topic | Page |
 |:---|:---|
+| Official real-project regression testing and PR comparison reports | [liveness_primer](./liveness-primer.md) |
 | Generated codebase navigator for contributors | [Skylos Repo Map](./repo-map/index.html) |
 | CLI output modes, pretty reports, and TUI controls | [CLI Output Modes](./cli-output.md) |
 | Optional Ruff Python linting through the Skylos CLI | [Python Linting](./python-linting.md) |

docs/liveness-primer.md

@@ -0,0 +1,123 @@
+# Real-project regression testing with liveness_primer
+
+[liveness_primer](https://github.com/mcdigman/liveness_primer), created and
+maintained by [Matthew Digman](https://github.com/mcdigman), is Skylos's official
+real-project regression testing tool. Its source, adapters, and pinned project
+corpus live in Matthew's repository. Skylos keeps the CI integration here.
+
+It answers a specific question: **what changes for real projects if we merge
+this PR?** It does not decide whether every finding is correct.
+
+## What CI runs
+
+The [Analyzer Blast Radius workflow](../.github/workflows/liveness-primer.yml)
+runs when a PR is opened, updated with new commits, reopened, or marked ready
+for review. There are no changed-path or draft filters: docs-only and draft
+PRs get the same comparison. Fork PRs may need maintainer approval to run, and
+GitHub cannot run this `pull_request` workflow while merge conflicts remain.
+See [GitHub's event documentation](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request).
+
+The primer builds two Skylos revisions in separate managed environments:
+
+- Base: the PR's exact base commit (`pull_request.base.sha`).
+- Head: GitHub's synthetic merge commit (`github.sha`), so the comparison
+  includes the result of merging the PR into its base branch, including forks.
+
+Both revisions scan the same commit-pinned corpus projects. The workflow uses
+the primer's packaged corpus, `--all`, two concurrent detector processes, and
+a 300-second default timeout per invocation. The job has a 45-minute limit.
+The primer and Actions are pinned to full commits; uv is pinned to `0.12.5`,
+Python to the `3.13` series, and the primer's dependencies use its lockfile.
+
+The current primer pin is
+[`d6f3118a2cfc465426500eab449005fe56845c58`](https://github.com/mcdigman/liveness_primer/tree/d6f3118a2cfc465426500eab449005fe56845c58).
+It selects 15 Python projects for Skylos. Its adapter compares unused
+functions, imports, classes, variables, and parameters. **File-level findings
+(`SKY-E002` / `SKY-E003`) are not included at this pin.** This job does not
+opt into security, secrets, quality, or AI-defect analyses, and is not a
+cross-language benchmark.
+
+## Read a PR report
+
+1. Open the PR's **Analyzer Blast Radius** run and read its summary.
+2. Check that both revisions completed. A crash, timeout, or unusable detector
+   output fails the job; it is not a clean comparison.
+3. Review added, dropped, and changed findings against the intent of the PR.
+   Added findings can be useful coverage or false positives. Dropped findings
+   can be fixed false positives or missed detections. Neither direction is
+   automatically an improvement.
+4. Download the `liveness-primer-report` artifact for
+   `liveness-primer-report.md` and the complete `liveness-primer-report.json`.
+   The Markdown display can be truncated; use JSON for the full comparison.
+   Artifacts are retained for 14 days, so save evidence needed for later work.
+
+Finding changes are advisory: the workflow does not use `--fail-on` gates.
+It preserves the primer's nonzero exit status and requires a nonempty JSON
+report. Available evidence is uploaded even if the comparison fails. A green
+check means the comparison completed, not that someone has approved its
+findings or proved the PR regression-free.
+
+If a change looks wrong, inspect the pinned source and turn the confirmed bug
+into a regression test or [Corpus Guard fixture](../corpus/README.md). Do not
+accept or reject a PR solely because its total finding count went down or up.
+
+## Reproduce a comparison
+
+Use a disposable Linux environment without credentials, with Git, Python 3.13,
+and uv 0.12.5 available. Managed runs build and execute the detector revisions;
+do not run an unfamiliar PR on your everyday development machine. Corpus
+projects are static-analysis inputs, not projects whose tests should be run.
+
+Clone the same primer revision:
+
+```bash
+git clone https://github.com/mcdigman/liveness_primer.git liveness-primer-check
+git -C liveness-primer-check checkout --detach d6f3118a2cfc465426500eab449005fe56845c58
+```
+
+Replace the two revision placeholders below with the full base and head SHAs
+from the report, not mutable branch names. The head is the reported merge SHA,
+not necessarily the PR branch tip.
+
+```bash
+uv run --project liveness-primer-check --python 3.13 --locked liveness-primer run \
+  --tool skylos \
+  --repo https://github.com/duriantaco/skylos \
+  --old BASE_SHA_FROM_REPORT \
+  --new MERGE_SHA_FROM_REPORT \
+  --all \
+  --output github \
+  --json-out liveness-primer-report.json \
+  --jobs 2 \
+  --timeout 300
+```
+
+The pinned primer requires enforced network isolation for managed runs on
+Linux and fails if it cannot establish it. CI uses a fresh GitHub-hosted
+runner, read-only repository permission, no passed secrets, no persisted
+checkout credentials, and no shared uv cache. Network isolation is not a
+complete filesystem sandbox; do not add credentials or use a persistent
+self-hosted runner for this job.
+
+The report records dependency versions and environment differences. A later
+run may resolve different detector dependencies even with the same source
+commits, so retain the original JSON when investigating a discrepancy.
+
+## Benchmarks and maintenance
+
+Primer reports can support change reviews and supply cases for the public
+[skylos-demo](https://github.com/duriantaco/skylos-demo) benchmarks. They do not
+supply ground-truth labels or precision/recall scores. Public accuracy claims
+still need labeled cases, pinned inputs, and the methodology in
+[BENCHMARK.md](../BENCHMARK.md). Corpus Guard and the existing benchmark gates
+remain in place.
+
+Keep corpus and adapter improvements upstream with Matthew, rather than
+copying the primer into Skylos. Propose threshold changes with examples and
+review them together; this integration does not impose new finding-count
+thresholds or transfer repository ownership.
+
+Update the primer pin in a separate reviewed change, run a fresh comparison,
+and inspect corpus, adapter, and report-schema changes before accepting it.
+Do not replace the pin with `main`. For example, upstream v0.1.1 adds file-level
+findings, so adopting it changes what this CI report covers.

test/test_liveness_primer_workflow.py

@@ -1,5 +1,11 @@
+import json
+import os
 from pathlib import Path
+import shutil
+import subprocess
+import sys
 
+import pytest
 import yaml
 
 from skylos.rules.config.cicd.github_actions import scan_github_actions_file
@@ -20,7 +26,7 @@ def _comparison_step(workflow):
     )
 
 
-def test_liveness_primer_workflow_is_scoped_read_only_and_advisory():
+def test_liveness_primer_workflow_covers_all_prs_read_only_and_advisory():
     workflow = _workflow()
     triggers = workflow.get("on", workflow.get(True))
 
@@ -32,20 +38,16 @@ def test_liveness_primer_workflow_is_scoped_read_only_and_advisory():
         "reopened",
         "ready_for_review",
     ]
-    assert set(pull_request["paths"]) == {
-        "skylos/**",
-        "pyproject.toml",
-        "MANIFEST.in",
-        ".github/workflows/liveness-primer.yml",
-    }
+    # Docs-only, packaging, tests, and fork PRs all need the same check.
+    assert set(pull_request) == {"types"}
     assert workflow["permissions"] == {"contents": "read"}
     assert workflow["concurrency"] == {
         "group": "liveness-primer-${{ github.event.pull_request.number }}",
         "cancel-in-progress": True,
     }
 
     job = workflow["jobs"]["blast-radius"]
-    assert job["if"] == "github.event.pull_request.draft == false"
+    assert "if" not in job  # Draft PRs get evidence too.
     assert job["runs-on"] == "ubuntu-24.04"
     assert job["timeout-minutes"] == 45
 
@@ -72,9 +74,7 @@ def test_liveness_primer_workflow_pins_actions_and_toolchain():
         assert all(character in "0123456789abcdef" for character in action_ref)
 
     checkout = next(
-        step
-        for step in steps
-        if step.get("name") == "Check out pinned liveness_primer"
+        step for step in steps if step.get("name") == "Check out pinned liveness_primer"
     )
     assert checkout["with"] == {
         "repository": "mcdigman/liveness_primer",
@@ -124,9 +124,7 @@ def test_liveness_primer_workflow_preserves_evidence_without_write_access():
     steps = workflow["jobs"]["blast-radius"]["steps"]
 
     artifact = next(
-        step
-        for step in steps
-        if step.get("name") == "Upload blast-radius evidence"
+        step for step in steps if step.get("name") == "Upload blast-radius evidence"
     )
     assert artifact["if"] == "always()"
     assert artifact["with"]["name"] == "liveness-primer-report"
@@ -142,3 +140,118 @@ def test_liveness_primer_workflow_preserves_evidence_without_write_access():
     assert "pull-requests: write" not in workflow_source
     assert "gh pr comment" not in workflow_source
     assert scan_github_actions_file(WORKFLOW_PATH, root=".") == []
+
+
+def _run_comparison(
+    tmp_path,
+    *,
+    exit_code=0,
+    report_state="present",
+    base_sha="a" * 40,
+    merge_sha="b" * 40,
+):
+    bash = shutil.which("bash")
+    if bash is None:
+        pytest.skip("workflow shell checks require bash")
+
+    # A shell function intercepts uv in the real workflow command. No primer,
+    # detector revisions, network requests, or corpus code are executed, and
+    # no executable fixture needs to be created on disk.
+    stub = """uv() {
+  "$PRIMER_STUB_PYTHON" - "$@" <<'PY'
+import json, os, sys
+with open('invocation.json', 'x', encoding='utf-8') as out:
+    json.dump(sys.argv[1:], out)
+if os.environ['PRIMER_STUB_REPORT'] != 'missing':
+    with open(os.environ['REPORT_JSON'], 'x', encoding='utf-8') as out:
+        if os.environ['PRIMER_STUB_REPORT'] == 'present':
+            json.dump({'fixture': 'offline workflow test'}, out)
+print('# Offline primer report')
+sys.exit(int(os.environ['PRIMER_STUB_EXIT']))
+PY
+}
+"""
+    env = {
+        "PATH": os.defpath,
+        "SKYLOS_REPOSITORY": "https://github.com/duriantaco/skylos",
+        "BASE_SHA": base_sha,
+        "MERGE_SHA": merge_sha,
+        # Spaces exercise quoting of the report destinations.
+        "REPORT_JSON": str(tmp_path / "report data.json"),
+        "REPORT_MARKDOWN": str(tmp_path / "report summary.md"),
+        "GITHUB_STEP_SUMMARY": str(tmp_path / "step summary.md"),
+        "PRIMER_STUB_EXIT": str(exit_code),
+        "PRIMER_STUB_REPORT": report_state,
+        "PRIMER_STUB_PYTHON": sys.executable,
+    }
+    return subprocess.run(
+        [bash, "-c", stub + _comparison_step(_workflow())["run"]],
+        cwd=tmp_path,
+        env=env,
+        capture_output=True,
+        text=True,
+        timeout=10,
+        check=False,
+    )
+
+
+def test_comparison_shell_passes_exact_revisions_and_keeps_both_reports(tmp_path):
+    result = _run_comparison(tmp_path)
+
+    assert result.returncode == 0, result.stderr
+    assert json.loads((tmp_path / "invocation.json").read_text()) == [
+        "run",
+        "--project",
+        "_liveness_primer",
+        "--locked",
+        "liveness-primer",
+        "run",
+        "--tool",
+        "skylos",
+        "--repo",
+        "https://github.com/duriantaco/skylos",
+        "--old",
+        "a" * 40,
+        "--new",
+        "b" * 40,
+        "--all",
+        "--output",
+        "github",
+        "--json-out",
+        str(tmp_path / "report data.json"),
+        "--jobs",
+        "2",
+        "--timeout",
+        "300",
+    ]
+    assert json.loads((tmp_path / "report data.json").read_text()) == {
+        "fixture": "offline workflow test"
+    }
+    assert (tmp_path / "report summary.md").read_text() == "# Offline primer report\n"
+    assert (tmp_path / "step summary.md").read_text() == "# Offline primer report\n"
+
+
+@pytest.mark.parametrize("exit_code", [1, 2, 3])
+def test_comparison_shell_does_not_hide_primer_failure_behind_tee(tmp_path, exit_code):
+    result = _run_comparison(tmp_path, exit_code=exit_code)
+
+    assert result.returncode == exit_code
+    assert (tmp_path / "report data.json").is_file()
+    assert (tmp_path / "report summary.md").read_text() == "# Offline primer report\n"
+
+
+@pytest.mark.parametrize("report_state", ["missing", "empty"])
+def test_comparison_shell_rejects_success_without_report_data(tmp_path, report_state):
+    result = _run_comparison(tmp_path, report_state=report_state)
+
+    assert result.returncode != 0
+    assert (tmp_path / "invocation.json").is_file()
+
+
+@pytest.mark.parametrize("revision", ["base_sha", "merge_sha"])
+def test_comparison_shell_rejects_non_commit_refs_before_running(tmp_path, revision):
+    result = _run_comparison(tmp_path, **{revision: "main"})
+
+    assert result.returncode != 0
+    assert "Invalid comparison revision" in result.stderr
+    assert not (tmp_path / "invocation.json").exists()