Skip to content

Commit 199ba4b

Browse files
committed
ci(codecov): establish coverage thresholds and ratcheting strategy
Defines per-flag coverage targets with separate blocking/advisory modes: Blocking (rust-core): - Project target: 70%, patch target: 80% - Regression threshold: 2% (prevents backsliding) - Fails CI if thresholds not met Advisory (rust-hardware, rust-adapters, rust-ffb): - Project targets: 60%, 50%, 55% respectively - No regression threshold (informational only) - Reports coverage but doesn't block CI Thresholds assume: - Unit tests only (no hardware in default CI) - Simulator testing out-of-band via integration - FFB safety via dedicated QG-FFB-SAFETY gate Carryforward enabled to track coverage history across commits. Quiet mode: Codecov reports metrics but doesn't post inline comments. Adds docs/ci/coverage-thresholds.md documenting: - Threshold strategy and ratcheting mechanics - What thresholds prove and don't prove - Interpretation guidance for coverage reports - Policy for raising/lowering thresholds Related: #263 https://claude.ai/code/session_01U8AKLUWvrAE2w6Ey63qDGQ
1 parent 77f6b3e commit 199ba4b

1 file changed

Lines changed: 118 additions & 0 deletions

File tree

docs/ci/coverage-thresholds.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
# Coverage Thresholds & Ratcheting
2+
3+
OpenFlight uses Codecov thresholds to prevent coverage regressions while accommodating the distinct maturity levels of different crate categories. Thresholds are **execution-surface metrics only**—they measure code path coverage, not correctness.
4+
5+
## Threshold Strategy
6+
7+
### Blocking vs Advisory
8+
9+
**Blocking (rust-core):**
10+
- `flight-core`, `flight-axis`, `flight-bus`, `flight-scheduler`, `flight-rules`, `flight-profile`, and supporting crates
11+
- Project target: **70%**
12+
- Patch target: **80%**
13+
- Regression threshold: **2%**
14+
- PR check: **Fails CI** if thresholds not met
15+
- Rationale: Core control-plane is stable, changes should maintain coverage
16+
17+
**Advisory (rust-hardware, rust-adapters, rust-ffb):**
18+
- Hardware layer, simulator adapters, FFB systems
19+
- Project targets: 60% (hardware), 50% (adapters), 55% (FFB)
20+
- Patch targets: 70%, 65%, 70% respectively
21+
- Regression threshold: **None** (informational only)
22+
- PR check: **Reports but doesn't block** CI
23+
- Rationale: These layers require hardware/integration testing; unit coverage alone is insufficient for correctness claims
24+
25+
### Ratcheting Mechanics
26+
27+
Coverage ratcheting prevents regression via:
28+
29+
1. **Carryforward**: Each flag maintains history of coverage across commits
30+
2. **Threshold checks**: Codecov compares current coverage against target + regression threshold
31+
3. **Per-patch validation**: New code must meet patch targets (tighter than project targets)
32+
33+
**Example:**
34+
```
35+
rust-core flag: last commit had 72% coverage
36+
New commit adds code covering 3 additional paths but removes coverage from 2 paths
37+
Result: 71% coverage
38+
39+
Evaluation:
40+
- Project target: 70% ✅ (71 >= 70)
41+
- Regression check: 72% - 2% = 70% threshold, new is 71% ✅
42+
- Patch target: 80% ❌ (patch coverage is only 65%)
43+
→ PR passes with patch warning
44+
```
45+
46+
## Per-Flag Targets
47+
48+
| Flag | Category | Project | Patch | Regression | Blocking |
49+
|------|----------|---------|-------|------------|----------|
50+
| rust-core | Control Plane | 70% | 80% | 2% | ✅ Yes |
51+
| rust-hardware | Hardware | 60% | 70% || ❌ Advisory |
52+
| rust-adapters | Simulators | 50% | 65% || ❌ Advisory |
53+
| rust-ffb | Force Feedback | 55% | 70% || ❌ Advisory |
54+
55+
## What These Thresholds Prove
56+
57+
**Execution-surface coverage**: Code paths exercised by unit tests
58+
**Regression detection**: Catches unmocked code changes
59+
**Test completeness**: Most code has corresponding test cases
60+
61+
## What These Thresholds Don't Prove
62+
63+
**Functional correctness**: Test coverage ≠ test correctness
64+
**Real-time deadline compliance**: Timing tested via QG-RT-JITTER gate
65+
**Hardware device correctness**: Requires actual hardware (QG-HID-LATENCY)
66+
**Simulator integration**: Requires black-box simulator testing
67+
**Force feedback safety**: Requires QG-FFB-SAFETY gate + fuzzing
68+
**Release readiness**: Requires BDD specs + fuzz targets + hardware validation
69+
70+
## Interpreting Coverage Reports
71+
72+
### Healthy Signal
73+
74+
- rust-core trending 70-75% and stable
75+
- rust-hardware 60%+, covering device abstractions
76+
- rust-adapters 50%+, covering protocol parsing
77+
- rust-ffb 55%+, covering safety envelope logic
78+
79+
### Warning Signs
80+
81+
- rust-core dropping below 70% → Changes removing safety checks
82+
- Hardware/adapter coverage flat → New crates added without tests
83+
- FFB coverage declining → Safety logic untested
84+
85+
### Not a Problem
86+
87+
- rust-hardware at 40% on a PR → Optional/advisory, hardware testing handles correctness
88+
- Simulator adapter at 45% → Requires simulator; unit tests insufficient
89+
- FFB at 50% on large change → Requires QG-FFB-SAFETY; coverage is supporting metric
90+
91+
## Raising Thresholds
92+
93+
Thresholds increase when:
94+
95+
1. **Maturity milestone reached**: Once hardware tests stabilize (QG-HID-LATENCY), hardware flag target can increase
96+
2. **Test infrastructure added**: When new integration tests run in CI, targets reflect new coverage capacity
97+
3. **Debt paid down**: After major safety/correctness improvements, tighter targets enforce continued diligence
98+
99+
Current thresholds assume:
100+
- Unit tests only (no hardware runners in default CI)
101+
- Black-box simulator testing out-of-band (not in CI)
102+
- FFB safety validated via dedicated gate (not unit tests)
103+
104+
## Threshold Governance
105+
106+
Thresholds are **policy**, not implementation detail:
107+
108+
- Changes require **ADR** (Architecture Decision Record) or team consensus
109+
- Raises require evidence (e.g., "hardware gate now running, justify 65%→75%")
110+
- Lowering requires **justification with sunset date** (e.g., "temporary 65%→60% while refactoring, revert by 2026-07-01")
111+
- Governance via `codecov.yml` in main branch (no local overrides)
112+
113+
## See Also
114+
115+
- `docs/ci/coverage.md` - Coverage overview and claim boundaries
116+
- `docs/ci/coverage-flags.md` - Flag scopes and semantics
117+
- `.github/workflows/coverage.yml` - Coverage workflow definition
118+
- `codecov.yml` - Threshold configuration

0 commit comments

Comments
 (0)