Skip to content

Commit 6218d38

Browse files
mihowclaude
andcommitted
feat(jobs): log per-job diagnostic on stalled-job revoke
When the reaper revokes a job, log a single WARN line capturing the state needed to triage "why was this stalled?" without grepping back through prior tick logs: - minutes since last update vs threshold - previous_status, dispatch_mode, celery_state - per-stage progress + status summary - pointer to running_job_snapshots for prior NATS consumer state Pairs with PR #1227's per-tick consumer snapshots: every 15 min, each running async_api job gets a NATS state snapshot; this new line tells operators which of those snapshots is the last one before revocation. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent f470bc6 commit 6218d38

1 file changed

Lines changed: 17 additions & 0 deletions

File tree

ami/jobs/tasks.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -454,6 +454,23 @@ def check_stale_jobs(minutes: int | None = None, dry_run: bool = False) -> list[
454454
job.finished_at = datetime.datetime.now()
455455
job.save()
456456
else:
457+
# Per-job diagnostic: surface enough state at revoke time that an
458+
# operator can answer "why was this stalled?" without grepping
459+
# back through tick logs. Pairs with the per-tick NATS consumer
460+
# snapshots logged by ``_run_running_job_snapshot_check``.
461+
stalled_minutes = (datetime.datetime.now() - job.updated_at).total_seconds() / 60
462+
stages_summary = (
463+
", ".join(f"{s.key}={s.progress*100:.1f}% {s.status}" for s in job.progress.stages)
464+
or "(no stages)"
465+
)
466+
job.logger.warning(
467+
f"Reaping stalled job: no progress for {stalled_minutes:.1f} min "
468+
f"(threshold {minutes} min). previous_status={previous_status}, "
469+
f"celery_state={celery_state}, dispatch_mode={job.dispatch_mode}, "
470+
f"stages: {stages_summary}. "
471+
f"For NATS consumer state at the last tick, see prior "
472+
f"running_job_snapshots logs for this job."
473+
)
457474
if not dry_run:
458475
job.update_status(JobState.REVOKED, save=False)
459476
job.finished_at = datetime.datetime.now()

0 commit comments

Comments
 (0)