R-3: Model-server crash recovery — respawn, fast-fail gate, local fallback

[TCVinNYC]

Summary

The opt-in shared-detector process (AUTOPTZ_MODEL_SERVER=1 — the validated 4-12-cam scaling mode) was spawn-once: if it died, every InferenceClient.detect() blocked 5 s and returned [] forever. Now:

• Supervisor respawn from the periodic health scan, mirroring the worker restart pattern exactly (exponential 1→30 s backoff, 5 attempts, reset-on-recovery). Respawn is non-blocking on the tick/GUI thread: start + deadline, then poll readiness on later ticks (adversarial review caught that a blocking ready.wait(30) would have frozen the Qt main thread up to 30 s per attempt; fixed with RED/GREEN evidence, independently reproduced by the re-reviewer).
• Same queues + shm slots reused across respawn (attach is lazy per-request — pinned by test); sequence-number stale-reply protection holds across the restart boundary; healthy clients unaffected.
• Client fast-fail: timeout_s 5→2 s default + a server_down gate that returns [] immediately during outages (no 2 s/frame crawl); gate stays set through spawning+backoff, cleared on ready.
• Budget exhaustion → visible fallback: model_server_failed() accessor (mirrors is_camera_failed()) + workers rebuild LOCAL detectors via RemotePool.detector() fallback, so detection continues degraded instead of dying.
• Model-server stays opt-in; no default flip; no knobs.

Test plan

• 19 new RED-first tests across tests/test_inference_server_ipc.py + tests/test_supervisor_health.py: kill-child→respawn→detections resume; immediate fast-fail during outage (wall-clock bound); budget exhaustion→flag+fallback; same-queue identity; non-blocking tick path (fake Event.wait raises on blocking call); spawn-deadline→failed-attempt backoff; behavioral 2 s timeout bound.
• Full suite 1936 passed; ruff + mypy + selftest green.
• Task review: two rounds (Needs-fixes → fixed → Approved; re-reviewer independently reproduced RED and GREEN).

Validation

CI-safe (opt-in path). Live mid-run server-kill exercise folds into the 2.2.0 soak gate. Includes the #141 test-isolation cherry-pick (dedupes on merge).

🤖 Generated with Claude Code