Cache split_tests pytest --collect-only results between CI runs#171780
Draft
bdraco wants to merge 33 commits into
Draft
Cache split_tests pytest --collect-only results between CI runs#171780bdraco wants to merge 33 commits into
bdraco wants to merge 33 commits into
Conversation
cProfile showed 99.6% of split_tests.py wall time was spent in the single pytest --collect-only subprocess. Fan out the collection across ``os.cpu_count()`` workers; round-robin chunking keeps each batch roughly equal, and tests/components is expanded one level deeper so the ~1000 integration subdirectories distribute evenly. Local wall time dropped from ~132s to ~11s on an 18-core box. Bucket output is unchanged because we still parse the same pytest -qq output, just aggregated from multiple invocations.
Only pass directories and test_*.py files to pytest --collect-only so helpers like tests/components/conftest.py and tests/components/common.py are not treated as explicit collection targets, and bail out with a clear error if no eligible paths are found instead of running pytest with no arguments.
Persist the result of pytest --collect-only between CI runs as a JSON file keyed by content hash, so unchanged test files are served from cache and only edited or new files are re-collected. The cache is self-healing: * Missing, corrupt, or wrong-version files fall back to a full collect. * Any conftest.py change anywhere under the test root invalidates the whole cache, so fixture parametrization shifts cannot silently skew counts. * Files pytest returns nothing for (helper modules named test_*.py with no test functions) are cached as zero so they don't get re-collected forever. Walking is done once with os.walk (~2x faster than Path.rglob) and collects test files plus conftests in a single pass. When the cache is fully cold we feed pytest top-level directories rather than thousands of file paths so cold runs stay as fast as before the cache landed. Wire the new --cache flag through the prepare-pytest-full job and back the cache file with actions/cache so PRs can pick up the latest dev snapshot via restore-keys. Local timings: cold 11s, warm with no diff 0.4s, warm with one file edited 2.3s.
This was referenced May 21, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an on-disk cache for script/split_tests.py so CI can reuse prior pytest --collect-only per-file counts and only re-collect changed/new test files, improving split-tests step performance across workflow runs.
Changes:
- Add JSON cache load/save + self-healing invalidation logic (schema version + conftest hash) and cache-aware collection flow in
script/split_tests.py. - Update CI workflow to restore/persist the cache file via
actions/cacheand pass--cachetoscript.split_tests. - Add comprehensive unit tests covering cache behavior, walker behavior, and cache hit/miss resolution.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
tests/script/test_split_tests.py |
Adds tests for the new cache + tree-walk logic and cache-driven collection behavior. |
script/split_tests.py |
Implements caching, conftest invalidation hashing, and cache-aware test collection. |
.github/workflows/ci.yaml |
Restores/saves the cache file in CI and passes it to split_tests.py via --cache. |
Address Copilot review feedback on the cache PR: * Split collect_tests into _collect_tests_uncached (the original directory-based pre-cache flow) and _collect_tests_cached (walks the tree to build per-file hashes). Without --cache we now skip the walk + per-file hash entirely. * A single-file root has no ancestor conftests to walk, so the conftest_hash would always be empty and stale counts could survive a real conftest change; bypass the cache for the file-root case. * Save the cache file with explicit utf-8 encoding and ensure_ascii=False.
This was referenced May 21, 2026
Member
Author
|
Still needs more work but basic idea is there and works |
Member
Author
|
@bluetoothbot review |
Comment on lines
+278
to
+285
| """Return ancestor conftests above ``root``, stopping at the first gap.""" | ||
| ancestors: list[Path] = [] | ||
| current = root.resolve().parent | ||
| while True: | ||
| conftest = current / "conftest.py" | ||
| if not conftest.is_file(): | ||
| break | ||
| ancestors.append(conftest) |
Comment on lines
+160
to
+166
| def test_find_ancestor_conftests_walks_up_until_gap(tmp_path: Path) -> None: | ||
| """Ancestor conftests are collected up to the first dir without one.""" | ||
| nested = tmp_path / "a" / "b" / "c" | ||
| nested.mkdir(parents=True) | ||
| # No conftest in tmp_path → walk stops there. | ||
| (tmp_path / "a" / "conftest.py").write_text("# a\n") | ||
| (tmp_path / "a" / "b" / "conftest.py").write_text("# b\n") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed change
Cache
pytest --collect-onlycounts between CI runs. The script writes a JSON file mapping eachtest_*.pyto its content hash, a fixture scope hash, and the collected test count; CI restores it withactions/cacheso unchanged files skip pytest collection.A file's fixture scope covers every
.pyon its ancestor path (conftests and helpers likecommon.py); this mirrors how pytest resolves fixtures and parametrize imports. Edits outside that ancestor chain leave the entry warm, so merging dev rarely busts the whole tree. Cold runs hand pytest the top-level directories instead of individual files, and when more than 30% of the tree misses we fall back to the same directory-level invocation. Same-directory test files normally stay in one bucket so syrupy can spot unused snapshots, but they're free to split when the folder has nosnapshots/directory.The cache self-heals on missing, corrupt, wrong-version, or malformed entries. In CI the save key is content-addressed (sha256 of the JSON file), so identical bucket inputs reuse the same actions cache entry instead of accumulating one per commit.
Type of change
Additional information
Checklist
ruff format homeassistant tests)If user exposed functionality or configuration variables are added/changed:
If the code communicates with devices, web services, or third-party tools:
Updated and included derived files by running:
python3 -m script.hassfest.requirements_all.txt.Updated by running
python3 -m script.gen_requirements_all.To help with the load of incoming pull requests: