Skip to content

fix(DB): check scheduled_task_runs independently in 0003 migration guard#4198

Open
DaoyuanLi2816 wants to merge 1 commit into
bytedance:mainfrom
DaoyuanLi2816:fix/migration-0003-idempotency-guard-both-tables
Open

fix(DB): check scheduled_task_runs independently in 0003 migration guard#4198
DaoyuanLi2816 wants to merge 1 commit into
bytedance:mainfrom
DaoyuanLi2816:fix/migration-0003-idempotency-guard-both-tables

Conversation

@DaoyuanLi2816

Copy link
Copy Markdown
Contributor

Summary

Revision 0003_scheduled_tasks creates both scheduled_tasks and scheduled_task_runs in one upgrade(), but its re-entry idempotency guard only checked the first table:

if inspector.has_table("scheduled_tasks"):
    return

On a DB that atypically already has scheduled_tasks but not its sibling scheduled_task_runs, the guard returns early and silently skips creating scheduled_task_runs. alembic_version still advances past 0003 (and on to head) with no error, so there's no retry path — the gap is permanent once triggered, not transient. The next code path that touches the table (e.g. ScheduledTaskRunRepository.count_active_runs()) then fails with OperationalError: no such table: scheduled_task_runs.

Honest scope note: this requires an atypical prior DB state

Reaching this bug requires the asymmetric state described above — scheduled_tasks present, scheduled_task_runs absent. The two ordinary bootstrap branches in persistence/bootstrap.py (empty-DB and legacy-DB) both provision the pair of tables atomically via Base.metadata.create_all, so a normal fresh install or normal upgrade never hits this. This is a genuine, verified violation of the code's own documented idempotency guarantee, relevant to restores, manual DB interventions, or partial-provisioning edge cases — not a claim of common/default-path impact.

Verification (real alembic chain, no mocks)

Per this repo's migration-authoring conventions (backend/AGENTS.md's Schema Migrations section), I verified against the real alembic chain rather than mocks:

  1. Seeded a fresh SQLite DB at revision 0002_runs_token_usage with only scheduled_tasks manually pre-created (simulating the atypical prior state).
  2. Ran alembic upgrade head (via the same init_enginebootstrap_schema path Gateway startup uses).
  3. Confirmed before the fix: scheduled_task_runs was never created, while alembic_version advanced to 0004_run_ownership anyway (no error raised) — and ScheduledTaskRunRepository.count_active_runs() raised OperationalError: no such table: scheduled_task_runs.
  4. Confirmed a second "restart" upgrade head does not self-heal it — the revision is already stamped as applied, so alembic never replays it.

Fix

Mirrors the per-object independent-check pattern already used in 0004_run_ownership.py's index creation (if "ix_runs_lease" not in existing:, checked separately per index). 0003_scheduled_tasks.py's upgrade() now snapshots has_table() for both tables up front and gates each op.create_table() independently, instead of gating the whole revision on just the first table's existence:

has_scheduled_tasks = inspector.has_table("scheduled_tasks")
has_scheduled_task_runs = inspector.has_table("scheduled_task_runs")

if not has_scheduled_tasks:
    op.create_table("scheduled_tasks", ...)
    ...

if not has_scheduled_task_runs:
    op.create_table("scheduled_task_runs", ...)
    ...

Testing

Added backend/tests/test_migration_0003_scheduled_task_runs_guard.py, following the same real-alembic-chain shape as test_migration_0004_run_ownership_dedupe.py:

  • test_scheduled_task_runs_created_when_only_scheduled_tasks_preexists — the bug scenario: seeds at 0002 with only scheduled_tasks, runs upgrade head, confirms scheduled_task_runs now gets created and a real repository call (count_active_runs()) succeeds instead of raising. Fails before the fix, passes after (verified via patch-revert: reverted the fix, reran, confirmed the same assertion fails with the same error; reapplied, reran, confirmed green).
  • test_neither_scheduled_table_exists_both_created_normally — ordinary path (neither table exists) still creates both.
  • test_both_scheduled_tables_already_exist_upgrade_head_is_idempotent — fully-idempotent path (both already exist, e.g. via the hybrid bootstrap's atomic create_all) still no-ops cleanly with no table ... already exists error.

Full local run (SQLite, real alembic chain, no mocks) — all passing:

  • test_migration_0003_scheduled_task_runs_guard.py (new, 3/3)
  • test_persistence_bootstrap.py (31/31)
  • test_persistence_bootstrap_concurrency.py
  • test_persistence_bootstrap_regression.py
  • test_persistence_bootstrap_pg_lock.py / test_persistence_bootstrap_sqlite_lock.py / test_persistence_bootstrap_url.py
  • test_persistence_migrations_env.py
  • test_migration_0004_run_ownership_dedupe.py
  • test_migration_user_isolation.py
  • test_scheduled_task_claims.py / test_scheduled_task_models.py / test_scheduled_task_repository.py / test_scheduled_task_schedules.py

ruff check --line-length 240 and ruff format --check --line-length 240 both pass repo-wide.

Before submitting

  • Regression test added, using the real alembic chain (no mocks)
  • Fail-before/pass-after verified via patch-revert
  • Full persistence/migrations/scheduled-task test suite passing locally
  • ruff check / ruff format --check clean (line-length 240)
  • Rebased onto latest main

Revision 0003_scheduled_tasks creates both scheduled_tasks and
scheduled_task_runs, but its re-entry guard only checked the first
table (inspector.has_table("scheduled_tasks")) before returning early.
A DB that atypically already has scheduled_tasks but not its sibling
scheduled_task_runs (a partial restore, manual DB surgery, or legacy
provisioning) silently skips creating scheduled_task_runs -- alembic_version
still advances past 0003 with no error, so a later restart's upgrade head
does not self-heal it.

Mirror the per-object independent-check pattern already used in
0004_run_ownership.py's index creation: check has_table() for both
tables up front and gate each op.create_table() independently, instead
of gating the whole revision on the first table's existence.
@github-actions github-actions Bot added area:backend Gateway / runtime / core backend under backend/ risk:medium Medium risk: regular code changes size/L PR changes 300-700 lines labels Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend Gateway / runtime / core backend under backend/ risk:medium Medium risk: regular code changes size/L PR changes 300-700 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant