Skip to content

Add catalog canary coverage audit#878

Merged
huangruiteng merged 1 commit into
mainfrom
codex/catalog-canary-coverage-audit-20260629
Jun 29, 2026
Merged

Add catalog canary coverage audit#878
huangruiteng merged 1 commit into
mainfrom
codex/catalog-canary-coverage-audit-20260629

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Jun 28, 2026

Copy link
Copy Markdown
Owner

Summary

  • add loopx canary coverage-audit to report P0/P1 catalog patterns missing family-profile coverage or explicit exception rationale
  • keep the command read-only: it does not execute smoke checks or create new runtime contracts
  • extend catalog canary planner smoke with real-catalog coverage, simulated matrix drift, deferred-owner exception, and CLI JSON checks

Validation

  • python3 examples/catalog-canary-planner-smoke.py
  • python3 -m compileall loopx/canary/planner.py loopx/cli_commands/canary.py
  • python3 -m loopx.cli --format json canary coverage-audit
  • python3 -m loopx.cli --format json check --scan-path loopx/canary/planner.py --scan-path loopx/cli_commands/canary.py --scan-path examples/catalog-canary-planner-smoke.py
  • git diff --check

Risk

Low-to-medium CLI/read-path addition. Existing canary profiles and canary plan behavior is unchanged; new output is opt-in under canary coverage-audit. The audit reports drift only and avoids forcing every pattern into one giant canary.

@huangruiteng

Copy link
Copy Markdown
Owner Author

No blocking findings from my side.

Product/architecture judgment:

  • Motivation: this PR tries to prevent the interaction-pattern catalog from drifting away from canary coverage by adding a read-only loopx canary coverage-audit surface for P0/P1 patterns.
  • Solved or not: the diff does solve the intended gap for the current catalog shape. It checks that selected catalog patterns are either covered by canary family profiles or explicitly excepted with rationale/deferred owner.
  • User/operator impact: maintainers get a lightweight drift report instead of manually scanning the catalog matrix or forcing every pattern into one giant canary profile.
  • Main risk: low to medium. It adds CLI/read-path behavior but does not execute checks, mutate state, or change existing canary profiles / canary plan behavior. The main compatibility risk is future catalog table-shape drift, which the smoke now covers with real-catalog and simulated-drift cases.
  • Design judgment: this is a reasonable reusable projection/audit surface rather than a one-off hardcoded report. Keeping it read-only and exception-table based is cleaner than expanding the default canary matrix indiscriminately.

Validation I ran locally on PR head 60104af3:

  • python3 examples/catalog-canary-planner-smoke.py
  • python3 -m compileall loopx/canary/planner.py loopx/cli_commands/canary.py
  • python3 -m loopx.cli --format json canary coverage-audit
  • python3 -m loopx.cli --format json check --scan-path loopx/canary/planner.py --scan-path loopx/cli_commands/canary.py --scan-path examples/catalog-canary-planner-smoke.py
  • git diff --check origin/main...HEAD

Merge decision: approved from side-agent review perspective, but I did not admin-bypass merge because this is a new CLI/read-path surface and the PR still has the normal review-required gate.

@huangruiteng huangruiteng force-pushed the codex/catalog-canary-coverage-audit-20260629 branch from 60104af to 56cb106 Compare June 29, 2026 13:41
@huangruiteng

Copy link
Copy Markdown
Owner Author

No blocking findings after rebasing onto current origin/main.

Product/architecture judgment:

  • Motivation: keep the interaction-pattern catalog and canary coverage from drifting apart by adding a read-only loopx canary coverage-audit surface for P0/P1 patterns.
  • Solved or not: solved for the current catalog/profile contract. The audit reports P0/P1 patterns missing profile coverage unless an explicit exception table supplies a valid rationale/owner, and the smoke covers both real-catalog and simulated-drift cases.
  • User/operator impact: maintainers get a cheap drift check before changing catalog/canary surfaces, without expanding default canaries into one large profile.
  • Main risk: low. This adds a read-only CLI/report path and smoke coverage; it does not execute checks, mutate state, change permission boundaries, or alter existing canary profiles, canary plan, or canary run behavior.
  • Design judgment: the exception-table based audit is a reusable contract surface and fits the catalog-informed canary design better than hardcoding one-off coverage assertions.

Validation on rebased head 56cb1069:

  • python3 examples/catalog-canary-planner-smoke.py
  • python3 examples/catalog-canary-run-e2e-smoke.py
  • python3 -m compileall loopx/canary/planner.py loopx/cli_commands/canary.py
  • python3 -m loopx.cli --format json canary coverage-audit
  • python3 -m loopx.cli --format json canary run --no-execute --changed-file loopx/cli_commands/canary.py --check-limit 2
  • python3 -m loopx.cli --format json check --scan-path loopx/canary/planner.py --scan-path loopx/cli_commands/canary.py --scan-path examples/catalog-canary-planner-smoke.py
  • git diff --check origin/main...HEAD

Public/private boundary scan: clean for the three changed files. The loopx check warning about duplicate loopx-meta runtime index rows is pre-existing local runtime state and unrelated to this PR.

Merge decision: self-merging with owner authorization.

@huangruiteng huangruiteng merged commit 699bde4 into main Jun 29, 2026
@huangruiteng huangruiteng deleted the codex/catalog-canary-coverage-audit-20260629 branch June 29, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant