Skip to content

perf(analysis): share parsed ASTs across rule passes - #779

Merged
duriantaco merged 2 commits into
duriantaco:mainfrom
mcdigman:codex/shared-ast-index-upstream
Sep 5, 2026
Merged

perf(analysis): share parsed ASTs across rule passes#779
duriantaco merged 2 commits into
duriantaco:mainfrom
mcdigman:codex/shared-ast-index-upstream

Conversation

@mcdigman

@mcdigman mcdigman commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Closes #775
Closes #778

Summary

  • Shares parsed Python ASTs across repo-wide analysis passes
  • reuses a per-tree AST index for phantom-reference and Python API-hallucination analysis, and memoizes scope information shared by both rule passes.
  • Fixes a bug where analysis silently aborted on an unparseable file, rather than continuing

Why?

The ai-defects rules contain 2 full parses and 4 full AST walks that are completely duplicative.

That could be improved in isolation, and indeed accounts for most of the perf improvement of this PR, but this PR proposes to build indices that unify all the AST walks in the same scope. In measured counts of ast.walk calls, with --ai-defects the PR reduces the total full AST walks from ~12 to ~2.

I can de-scope unifying all the other AST walks if desired.

As implemented, the fix would inevitably create a merge conflict with an isolated fix to #775, so I just incorporated a fix to that into this PR as well; it could be done as a separate PR that then has its code overwritten by this one, or fixed in isolation if the full unification is not desired.

Also, as implemented, the cache is not thread safe if the outer analyze was called separately from multiple different threads. That isn't how skylos uses it so I thought that was an acceptable caveat, but it can be fixed or modified if desired by the maintainer; I didn't want to get too far ahead without knowing if merging all the other AST.walks outside ai-defects is ok in the first place.

AI Use Disclosure
The duplicate walks were found by analysis of cProfile results, and scoped the way to unify them. Opus 5.0 drafted the changes and they were reviewed according to skylos repository standards by codex GPT-5.6 Sol and claude Fable 5.

Performance impact

About a ~10% reduction in overall time for liveness_primer scans including --ai-defects and --quality. In the mcp-context-forge example, the biggest component of the perf impact is shaving ~30s from the ai-defects scan. For mcp-context-forge, cuts ~31 million calls to ast.walk.

How to test

  • python -m pytest -q test/test_analyzer.py test/test_api_signature_hallucination.py test/test_python_api_hallucination.py test/test_dead_code.py test/test_dead_code_liveness.py
  • python -m pytest -q test/
  • python scripts/corpus_ci.py --manifest corpus/manifest.json

Results on this head:

@mcdigman
mcdigman marked this pull request as ready for review August 28, 2026 22:23
@mcdigman
mcdigman requested a review from duriantaco as a code owner August 28, 2026 22:23

@duriantaco duriantaco left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@duriantaco
duriantaco merged commit f99402e into duriantaco:main Sep 5, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Perf: duplicated parses and ast walks SKY-L012/SKY-L023: Non-parseable file causes scan loop to silently abort

2 participants