feat(BA-6242): give custom runtime variant a default health check#11863
Merged
Conversation
github-actions
Bot
added
size:L
seedspirit
added a commit
that referenced
this pull request
May 29, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Seeds the custom runtime variant’s default_model_definition with a baseline health_check so deployments that omit health_check don’t inherit the strict 60s schema default and get marked unhealthy while large models are still loading.
Changes:
- Add an Alembic data migration to update the
customruntime variant’sdefault_model_definitionfrom{"models": null}to a draft containingservice.health_checkdefaults (guarded to avoid overwriting operator customizations). - Update both runtime-variant fixture JSON files to match the new
customdefault. - Add a towncrier news fragment describing the behavior change.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/ai/backend/manager/models/alembic/versions/ed42bc179b91_set_custom_runtime_variant_default_definition.py |
Adds an idempotent, guarded UPDATE to seed custom’s default model-definition draft with a long initial_delay health check. |
src/ai/backend/install/fixtures/example-runtime-variants.json |
Updates the installer fixture so fresh installs include the new custom baseline health check. |
fixtures/manager/example-runtime-variants.json |
Keeps the Manager-side fixture in sync with the installer fixture for the custom default. |
changes/11863.feature.md |
Documents the new default health check behavior in release notes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1 @@ | |||
| Apply a default health check (/health, 1800s initial delay) to the custom runtime variant so model deployments no longer fail health checks while large models load | |||
fregataa
reviewed
May 29, 2026
Member
fregataa
left a comment
fregataa
left a comment
There was a problem hiding this comment.
health check field is nullable and when it is null then no health check runs
Seed the custom variant's default_model_definition with a baseline health_check (path /health, initial_delay 1800s) so deployments that omit it no longer fall back to the 60s schema default and fail health checks while large models load. User model-definition.yaml / request values still override field-by-field. Migration + both fixtures kept in sync. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…flag Add an `enable` flag (default False) to ModelHealthCheck across the config model, v2 DTOs, GraphQL type/input, adapter, and runtime-variant fixtures. health_check_config() now gates on `enable` instead of presence, and a model-definition file that declares a health_check is treated as enabled via ModelDefinitionDraft.from_file_payload. Re-parent the custom runtime variant seed migration onto eb9d9c018e85 and seed a disabled default health check so custom deployments can opt in later. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: octodog <mu001@lablup.com>
fregataa
reviewed
Jun 4, 2026
| if not changed: | ||
| return draft | ||
| return draft.model_copy(update={"models": new_models}) | ||
|
|
Per review, the creation input boolean does not need to be nullable. ModelHealthCheckInput.enable and ModelHealthCheckInputGQL.enable now default to False instead of None. The draft type stays nullable so merge semantics are unaffected (to_draft uses exclude_unset). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…licit opt-out
An empty `health_check:` (null or {}) in a model-definition.yaml now resolves to
a disabled override instead of leaking the runtime-variant/preset baseline. A
non-empty block still opts in (enable defaults to True). Normalization runs on
the canonical-name dump so it is correct for both snake_case and kebab-case keys.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
HyeockJinKim
requested changes
Jun 5, 2026
…riants Built-in variants ship a known health_check endpoint, so they should be health-checked by default (enable=true); only `custom` stays opt-in (false). Corrects the example fixtures and folds the existing-DB backfill into the ed42bc179b91 health-check opt-in migration. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
HyeockJinKim
approved these changes
Jun 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
customruntime variant'sdefault_model_definitionwith a baselinehealth_check(path/health,interval10s,max_retries10,initial_delay1800s) via alembic migrationed42bc179b91.customheld{"models": null}, so deployments that omittedhealth_checkfell back to the 60s schema default and could be marked unhealthy while large models load (up to ~30 min). 1800s matches the prebuilt variants.model-definition.yaml/ request still overrides each field, so this is purely a fallback layer. Operator-customised rows are left untouched (WHERE-guarded UPDATE, idempotent).fixtures/manager/andsrc/ai/backend/install/fixtures/) in sync with the migration.Test plan
pants fmt/lint/checkon the migration0113c63f3261 → ed42bc179b91seeds the def;downgrade -1reverts to{"models": null}; re-upgrade headis idempotentResolves BA-6242
📚 Documentation preview 📚: https://sorna--11863.org.readthedocs.build/en/11863/
📚 Documentation preview 📚: https://sorna-ko--11863.org.readthedocs.build/ko/11863/