Name of failing test
Repo-wide — not a single test. Observed across unrelated suites in build 78354 (PR #48150): basic-correctness, language-models-tests-standard, entrypoints-integration-api-server-generate, lm-eval-kv-offload-1xh200, rust-frontend-distributed, metrics-tracing-2-gpus, distributed-*, AMD jobs, and more (13 failed at time of filing, more pending jobs likely to follow).
Basic information
🧪 Describe the failing test
Every failed job log in the build is saturated with HuggingFace Hub 504 Gateway Timeout errors on model downloads and metadata requests (12–228 occurrences per log), e.g.:
HTTP Error 504 thrown while requesting HEAD https://huggingface.co/nvidia/Nemotron-H-8B-Base-8K/resolve/main/processor_config.json
Retrying in 8s [Retry 4/5].
ERROR [repo_utils.py:66] Error retrieving file list: Server error '504 Gateway Timeout' for url 'https://huggingface.co/api/models/google/gemma-4-E4B-it/tree/main?...'
Servers/engines that don't need fresh downloads come up fine; jobs then stall on Hub retries until the harness timeout kills them. The failure window starts around 2026-07-16 07:50 UTC.
No action needed on any individual PR — failed jobs just need a retry once the Hub recovers. Filing this so other PRs hitting the same reds can dedupe here.
Possible hardening (optional follow-up)
CI runners already mount a shared HF_HOME cache, but jobs still issue Hub HEAD/metadata requests for models that are fully cached (see the processor_config.json HEAD retry loop above). Setting HF_HUB_OFFLINE=1 for jobs whose models are pre-cached — or falling back to the local cache on 5xx — would make CI resilient to Hub outages like this one.
Name of failing test
Repo-wide — not a single test. Observed across unrelated suites in build 78354 (PR #48150):
basic-correctness,language-models-tests-standard,entrypoints-integration-api-server-generate,lm-eval-kv-offload-1xh200,rust-frontend-distributed,metrics-tracing-2-gpus,distributed-*, AMD jobs, and more (13 failed at time of filing, more pending jobs likely to follow).Basic information
🧪 Describe the failing test
Every failed job log in the build is saturated with HuggingFace Hub
504 Gateway Timeouterrors on model downloads and metadata requests (12–228 occurrences per log), e.g.:Servers/engines that don't need fresh downloads come up fine; jobs then stall on Hub retries until the harness timeout kills them. The failure window starts around 2026-07-16 07:50 UTC.
No action needed on any individual PR — failed jobs just need a retry once the Hub recovers. Filing this so other PRs hitting the same reds can dedupe here.
Possible hardening (optional follow-up)
CI runners already mount a shared
HF_HOMEcache, but jobs still issue Hub HEAD/metadata requests for models that are fully cached (see theprocessor_config.jsonHEAD retry loop above). SettingHF_HUB_OFFLINE=1for jobs whose models are pre-cached — or falling back to the local cache on 5xx — would make CI resilient to Hub outages like this one.