|
| 1 | +# Coverage Thresholds & Ratcheting |
| 2 | + |
| 3 | +OpenFlight uses Codecov thresholds to prevent coverage regressions while accommodating the distinct maturity levels of different crate categories. Thresholds are **execution-surface metrics only**—they measure code path coverage, not correctness. |
| 4 | + |
| 5 | +## Threshold Strategy |
| 6 | + |
| 7 | +### Blocking vs Advisory |
| 8 | + |
| 9 | +**Blocking (rust-core):** |
| 10 | +- `flight-core`, `flight-axis`, `flight-bus`, `flight-scheduler`, `flight-rules`, `flight-profile`, and supporting crates |
| 11 | +- Project target: **70%** |
| 12 | +- Patch target: **80%** |
| 13 | +- Regression threshold: **2%** |
| 14 | +- PR check: **Fails CI** if thresholds not met |
| 15 | +- Rationale: Core control-plane is stable, changes should maintain coverage |
| 16 | + |
| 17 | +**Advisory (rust-hardware, rust-adapters, rust-ffb):** |
| 18 | +- Hardware layer, simulator adapters, FFB systems |
| 19 | +- Project targets: 60% (hardware), 50% (adapters), 55% (FFB) |
| 20 | +- Patch targets: 70%, 65%, 70% respectively |
| 21 | +- Regression threshold: **None** (informational only) |
| 22 | +- PR check: **Reports but doesn't block** CI |
| 23 | +- Rationale: These layers require hardware/integration testing; unit coverage alone is insufficient for correctness claims |
| 24 | + |
| 25 | +### Ratcheting Mechanics |
| 26 | + |
| 27 | +Coverage ratcheting prevents regression via: |
| 28 | + |
| 29 | +1. **Carryforward**: Each flag maintains history of coverage across commits |
| 30 | +2. **Threshold checks**: Codecov compares current coverage against target + regression threshold |
| 31 | +3. **Per-patch validation**: New code must meet patch targets (tighter than project targets) |
| 32 | + |
| 33 | +**Example:** |
| 34 | +``` |
| 35 | +rust-core flag: last commit had 72% coverage |
| 36 | +New commit adds code covering 3 additional paths but removes coverage from 2 paths |
| 37 | +Result: 71% coverage |
| 38 | +
|
| 39 | +Evaluation: |
| 40 | +- Project target: 70% ✅ (71 >= 70) |
| 41 | +- Regression check: 72% - 2% = 70% threshold, new is 71% ✅ |
| 42 | +- Patch target: 80% ❌ (patch coverage is only 65%) |
| 43 | +→ PR passes with patch warning |
| 44 | +``` |
| 45 | + |
| 46 | +## Per-Flag Targets |
| 47 | + |
| 48 | +| Flag | Category | Project | Patch | Regression | Blocking | |
| 49 | +|------|----------|---------|-------|------------|----------| |
| 50 | +| rust-core | Control Plane | 70% | 80% | 2% | ✅ Yes | |
| 51 | +| rust-hardware | Hardware | 60% | 70% | — | ❌ Advisory | |
| 52 | +| rust-adapters | Simulators | 50% | 65% | — | ❌ Advisory | |
| 53 | +| rust-ffb | Force Feedback | 55% | 70% | — | ❌ Advisory | |
| 54 | + |
| 55 | +## What These Thresholds Prove |
| 56 | + |
| 57 | +✅ **Execution-surface coverage**: Code paths exercised by unit tests |
| 58 | +✅ **Regression detection**: Catches unmocked code changes |
| 59 | +✅ **Test completeness**: Most code has corresponding test cases |
| 60 | + |
| 61 | +## What These Thresholds Don't Prove |
| 62 | + |
| 63 | +❌ **Functional correctness**: Test coverage ≠ test correctness |
| 64 | +❌ **Real-time deadline compliance**: Timing tested via QG-RT-JITTER gate |
| 65 | +❌ **Hardware device correctness**: Requires actual hardware (QG-HID-LATENCY) |
| 66 | +❌ **Simulator integration**: Requires black-box simulator testing |
| 67 | +❌ **Force feedback safety**: Requires QG-FFB-SAFETY gate + fuzzing |
| 68 | +❌ **Release readiness**: Requires BDD specs + fuzz targets + hardware validation |
| 69 | + |
| 70 | +## Interpreting Coverage Reports |
| 71 | + |
| 72 | +### Healthy Signal |
| 73 | + |
| 74 | +- rust-core trending 70-75% and stable |
| 75 | +- rust-hardware 60%+, covering device abstractions |
| 76 | +- rust-adapters 50%+, covering protocol parsing |
| 77 | +- rust-ffb 55%+, covering safety envelope logic |
| 78 | + |
| 79 | +### Warning Signs |
| 80 | + |
| 81 | +- rust-core dropping below 70% → Changes removing safety checks |
| 82 | +- Hardware/adapter coverage flat → New crates added without tests |
| 83 | +- FFB coverage declining → Safety logic untested |
| 84 | + |
| 85 | +### Not a Problem |
| 86 | + |
| 87 | +- rust-hardware at 40% on a PR → Optional/advisory, hardware testing handles correctness |
| 88 | +- Simulator adapter at 45% → Requires simulator; unit tests insufficient |
| 89 | +- FFB at 50% on large change → Requires QG-FFB-SAFETY; coverage is supporting metric |
| 90 | + |
| 91 | +## Raising Thresholds |
| 92 | + |
| 93 | +Thresholds increase when: |
| 94 | + |
| 95 | +1. **Maturity milestone reached**: Once hardware tests stabilize (QG-HID-LATENCY), hardware flag target can increase |
| 96 | +2. **Test infrastructure added**: When new integration tests run in CI, targets reflect new coverage capacity |
| 97 | +3. **Debt paid down**: After major safety/correctness improvements, tighter targets enforce continued diligence |
| 98 | + |
| 99 | +Current thresholds assume: |
| 100 | +- Unit tests only (no hardware runners in default CI) |
| 101 | +- Black-box simulator testing out-of-band (not in CI) |
| 102 | +- FFB safety validated via dedicated gate (not unit tests) |
| 103 | + |
| 104 | +## Threshold Governance |
| 105 | + |
| 106 | +Thresholds are **policy**, not implementation detail: |
| 107 | + |
| 108 | +- Changes require **ADR** (Architecture Decision Record) or team consensus |
| 109 | +- Raises require evidence (e.g., "hardware gate now running, justify 65%→75%") |
| 110 | +- Lowering requires **justification with sunset date** (e.g., "temporary 65%→60% while refactoring, revert by 2026-07-01") |
| 111 | +- Governance via `codecov.yml` in main branch (no local overrides) |
| 112 | + |
| 113 | +## See Also |
| 114 | + |
| 115 | +- `docs/ci/coverage.md` - Coverage overview and claim boundaries |
| 116 | +- `docs/ci/coverage-flags.md` - Flag scopes and semantics |
| 117 | +- `.github/workflows/coverage.yml` - Coverage workflow definition |
| 118 | +- `codecov.yml` - Threshold configuration |
0 commit comments