Skip to content

Add new rule for suspicious regex false match/hallucinated dependency bumps #797

Description

@mcdigman

What problem does this solve?
Regular expression scans or AI searches during version number bump PRs can falsely match and advance the pinned version numbers of unrelated dependencies at the same time; see e.g. #790, and the other candidate historical examples in skylos below in 'Additional Context'. I have also committed or almost committed bumps with this exact behavioral mistake.

This rule would have separate value from the 'hallucinated dependency' rules like SKY-D222/SKY-D225, because a dependency floor pump could be undesirable/break backward compatibilities even if it is to a version number that actually does exist.

In certain cases some e.g. ruff rules (for example I think some scanning numpy behavior) are version floor sensitive, so hallucinated version bumps could also change the behavior of other scanners or result in compatibility-breaking automated fixes.

Proposed solution
Implement a new advisory SKY-AXXX (SKY-A106?) rule that scans PR diffs for cases where a version number update for the core repo is exactly mirrored by a version number update in a dependency. It is an advisory diff rule because it structurally can't have 100% precision, but in many repos version number update PRs would be separate from e.g. dependency bump/dependabot PRs, so the risk of excessive false positives is probably low.

Message could be something like "Suspicious dependency bump: dependency version number change exactly mirrors repository version number change."

Alternatives considered
Not implementing the rule.

Additional context
Codex scan for potential instances of this exact error in skylos turns up several candidates for the exact behavior this rule would be designed to catch. Presented below; I vetted the list and removed two questionable examples that my proposed rule wouldn't have matched.

Skylos transition Mirrored dependency change Assessment
2.0.0 → 2.0.1 → 2.1.0 → 2.1.1 Flask followed >=2.0.0 → >=2.0.1 → >=2.1.0 → >=2.1.1 in setup.py and pyproject.toml. Suspicious: these Flask releases were already years old. See b751cb7, and d054b1d.
3.0.0 → 3.0.1 → 3.0.2 → 3.0.3 flask-cors followed exactly, alongside inquirer. Almost certainly mechanical replacement.
3.0.0 → 3.0.1 → 3.0.2 → 3.0.3 markdown-it-py in uv.lock followed exactly. Definitely corrupt: PyPI has 3.0.0 but no 3.0.1–3.0.3, while the lock retained the 3.0.0 hashes and merely rewrote the filenames. See the PyPI release history.
3.1.3 → 3.2.0 → 3.2.1 → 3.2.2 → 3.2.3 Flask’s locked version followed exactly. Definitely corrupt: Flask’s real 3.1.3 artifact URLs and hashes were repeatedly relabeled as nonexistent 3.2.x releases. See the Flask release history and commits e891114 through 194c1f1.
3.4.0 → 3.4.1 jaraco-classes became 3.4.1. Definitely corrupt: only 3.4.0 exists, and the lock retained its hashes. See the PyPI release history and 22df997.
3.4.1 → 3.4.2 The inquirer lock entry became 3.4.2, while the Keyring exclusion changed from !=3.4.1 to !=3.4.2. Definitely accidental: Inquirer’s URLs still named 3.4.1, and both Keyring exclusions were already impossible under keyring>=25.6.0. See 178306b.

The final case was immediately followed by commits named fix: changed inquirer back to 3.4.1 and fix: regenerated corrupted uv lock.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions