(POC) Investigate uv by JacobCoffee · Pull Request #19953 · pypi/warehouse

JacobCoffee · 2026-04-27T23:05:43Z

Very POC to see how well we could fare from having modern tooling that wasn't super slow in local development.

Test swappin pip -> uv for local dev. Could also try pixi or something else with the uv resolver backend like pdm but was most familiar with this and we did the whole ruff-ification recently.

Initial commit (4e424ba) uses same requirements/*.txt format, no lockfile change for now. Would prefer modern pyproject.toml + lockfile if we were to proceed. Later commits modernizes further. but you can just checkout the initial.

Cold-cache, no-docker-cache benchmarks on my machine:

What	pip / pip-compile	uv	speedup
`make deps_upgrade_all` (resolve all 8 .in files)	83.7s	7.4s	11.3x
└ `main.in` alone	39.2s	1.9s	20.3x
`make build` base image — install steps	52.3s	10.0s	5.2x
`make build` docs image — install steps	11.4s	5.5s	2.1x
Python install per teardown+rebuild (base+docs)	92.3s	74.6s	−17.7s

bottleneck here is the frontend stuff, could swap to rspack

history here is pretty linear.

4e424ba will work standalone (not ci) to see just the uv change
958e1c2 swapps to pyproject toml and lock file
ed45666 swaps to docker stuff to frozen sync w/ groups trying to address @dstufft's PyPI circular-dep concern
b76e18f domt forge tto relock but really we should NEVER have to do that bc uv tooling ensure relocks but "just in case" someone edits pyproject.toml manually ig uess?
78f3a53 for removing old style requirements in favor of pyproj. toml

Port the contents of requirements/*.in into a single PEP 735 [dependency-groups] table in pyproject.toml. main.in becomes the [project].dependencies; the seven other .in files become groups (deploy, dev, ipython, lint, tests, docs-dev, docs-user, docs-blog). Generate uv.lock alongside (324 packages, 1,755 SHA-256 hashes, single cross-platform lock vs the previous 8 per-arch .txt files). tool.uv.package = false marks warehouse as an application, not a library — uv sync will not try to build a wheel. Existing requirements/*.txt remain in place; they are removed in a later commit once the Dockerfile + Makefile have switched over to uv sync --frozen.

Replace the per-stage uv pip install -r requirements/*.txt calls with uv sync --frozen reading directly from pyproject.toml + uv.lock. The docs stage now bind-mounts pyproject.toml + uv.lock instead of the requirements/ directory; the build stage collapses three RUN steps (dev, ipython, main+deploy+tests+lint) into a single uv sync that conditionally adds groups based on $DEVEL, $IPYTHON, $CI. Add UV_PROJECT_ENVIRONMENT=/opt/warehouse so uv sync targets the venv we created (it does not respect VIRTUAL_ENV for project syncs). --no-default-groups disables uv's auto-installation of the dev group. Side benefits: - Reads URLs + SHA-256s straight from uv.lock; no PyPI index lookups during install (resolves the warehouse-deploys-on-warehouse circular dependency dstufft has flagged). - Combined install drops from 18.3s (3x uv pip install) to 10.4s. - Removes ~15 lines of Dockerfile. requirements/*.txt remain on disk for now; bin/deps* still consume them and will be removed in a follow-up commit.

Catches the case where someone edits pyproject.toml dependencies but forgets to regenerate uv.lock. Runs first in bin/lint so it fails fast before the longer ruff/mypy/djlint passes.

pyproject.toml + uv.lock are now the sole source of truth for Python dependencies; the Dockerfile, Makefile, and CI cache key all read from them directly. The requirements/ workflow (and the bin/deps, bin/deps-upgrade, bin/depchecker.py scripts that drove it) is no longer reachable. Removed: - requirements/ directory (8 .in files + 9 .txt files, ~6700 lines) - requirements.txt (root dependabot shim) - bin/deps, bin/deps-upgrade, bin/depchecker.py - pip-tools and pip-api from the dev dependency group - [tool.pip-tools.compile] from pyproject.toml Updated: - Makefile: deps / deps_upgrade_all / deps_upgrade_project now run 'uv lock --check' / 'uv lock --upgrade' / 'uv lock --upgrade-package' instead of bin/deps* scripts. Pattern rule for requirements/%.txt is gone. - Makefile: .state/docker-build-{base,docs} dependencies updated from requirements/*.txt to pyproject.toml + uv.lock. - .github/workflows/ci.yml: mypy cache key hashes pyproject.toml + uv.lock instead of requirements/*.txt. Verified make build (no-cache) still succeeds in ~55s.

…lock --check The migration to pyproject.toml + uv.lock allowed uv to resolve forward to newer versions than the previous requirements/*.txt files held; that caused real breakage in CI's Check Database Consistency job: AttributeError: 'SchemaPath' object has no attribute 'contents' Triggered by openapi-core 0.22 → 0.23 + jsonschema-path 0.3 → 0.4, which moved an attribute pyramid_openapi3 still calls. The migration policy should be: tooling change only, no functional dep diffs. Future bumps are deliberate per-package PRs. This commit: - Adds tool.uv.constraint-dependencies (299 entries) pinning every transitive to the version it had in the deleted requirements/*.txt files. Verified zero drift on shared packages. - Adds tool.uv.environments = ["sys_platform == 'linux'"] to scope the lock to the single platform we deploy to. The previous pip-compile workflow was implicitly single-platform; uv's default cross-platform locking surfaced win32 marker conflicts (e.g. mkdocs-rss-plugin 1.17.9 pinning tzdata<2026 on win32) that were silently ignored before. Re-enabling cross-platform is a follow-up. - Updates .github/workflows/ci.yml: the Dependencies job ran bin/deps, which the cleanup commit deleted. Replaced with uv lock --check (the new equivalent — same drift-detection purpose). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…lint The Dependencies job ran 'uv lock --check' (was 'bin/deps' pre-migration) to verify pyproject.toml/uv.lock are consistent. That same check now runs as the first line of bin/lint (added in commit b76e18f), so the Lint job already covers it. Keeping a separate matrix entry just duplicated the check and required adding uv to the runtime container or restructuring the job to run outside the container; both costs for no extra signal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

bin/rtd-docs ran 'pip install -r requirements/docs-{dev,user,blog}.txt' to install Read the Docs build deps. Those files were removed in commit 78f3a53; RTD broke for all three projects (warehouse, docspypiorg, blogpypiorg). Replace with: - 'pip install uv' so uv is available in RTD's host Python (RTD does not run our Dockerfile) - 'uv sync --frozen --no-default-groups --group docs-X' to install just the relevant docs group from pyproject.toml + uv.lock - 'uv run --no-sync mkdocs build ...' so mkdocs runs in the synced venv without re-resolving Also update the change-detection diff filter: it watched 'requirements/docs-X.txt' for skip-if-unchanged decisions; replace with 'pyproject.toml uv.lock'. This over-triggers slightly (any dep change busts the skip) but is the simplest approximation; finer per-group detection would need to parse the dep groups out of pyproject. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

bin/lint now runs 'uv lock --check' as its first step. The Lint CI job runs bin/lint inside the deployed runtime container, which previously had no uv on PATH (uv was only copied into the build/docs stages). Result: 'uv: command not found' / exit 127. Add the same COPY --from=ghcr.io/astral-sh/uv to the final stage. ~25MB image growth; keeps uv available for any in-container task (lint, deps upgrades, ad-hoc shell work). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… to bin/lint" This reverts the matrix removal. Dependencies is a required check in branch protection, so removing it leaves the PR's "Dependencies" status stuck on "Expected — Waiting for status to be reported" indefinitely. uv is now in the runtime stage (commit daadb0e), so the same 'uv lock --check' that bin/lint runs as its first step works fine as a standalone matrix entry too. Slight duplication, much simpler than either removing the branch protection requirement or restructuring. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

JacobCoffee · 2026-04-28T00:13:57Z

we continued poking at this, to see what ci and other things would look like.

CI wall-clock comparison: pre-ruff baseline → ruff PR → uv migration

GitHub Actions, same workflow (.github/workflows/ci.yml), same depot-ubuntu-24.04-arm runners. All three runs ran the same set of jobs and all passed.

Job	#19941 (pre-ruff)	#19943 (ruff merged)	#19953 (uv migration)	uv vs pre-ruff
build (Docker image)	197 s	173 s	71 s	−64% / 2.77× faster
Tests	244 s	228 s	211 s	−14%
Lint	100 s	65 s	62 s	−38%
Check DB Consistency	99 s	86 s	78 s	−21%
User Documentation	85 s	52 s	51 s	−40%
Developer Documentation	69 s	52 s	52 s	−25%
Licenses	69 s	64 s	52 s	−25%
Translations	86 s	52 s	53 s	−38%
Dependencies	95 s	76 s	50 s	−47%
Total wall (build + slowest matrix)	~441 s (7m21s)	~401 s (6m41s)	~282 s (4m42s)	−36% / 1.56× faster

The matrix jobs run in parallel, so total wall = build + slowest matrix job (Tests).

Where the time went:

Ruff alone (Use ruff to format and lint #19943): ~24 s off build (smaller lint deps to install) and ~35 s off Lint (one tool replaces four). Total wall: ~40 s.
uv on top ((POC) Investigate uv #19953): another ~102 s off build (uv sync --frozen reads URLs+hashes from the lock — no PyPI index lookups, no resolver round-trip), ~17 s off Tests (faster image = faster job startup), ~26 s off Dependencies (uv lock --check is sub-second vs the old bin/deps). Total wall: another ~119 s.

Stacked: ~159 seconds saved per CI run vs pre-ruff. PRs running CI 5+ times across review iterations recover ~13 minutes of wall-clock per PR.

dstufft · 2026-04-30T16:36:16Z

+    "types-zxcvbn==4.5.0.20260408",
+    "typing-extensions==4.15.0",
+    "typing-inspection==0.4.2",
+    "tzdata==2026.1",


Suggested change

"tzdata==2026.1",

If you delete this and re-run uv lock, it'll select one or two versions back and then will work on windows as well. Not the biggest deal either way, but nice to support

dstufft · 2026-04-30T16:37:03Z

+name = "warehouse"
+version = "0.0.0"
+description = "PyPI's web application"
+requires-python = ">=3.13"


Suggested change

requires-python = ">=3.13"

requires-python = "==3.13.*"

we only support a single version at a time, so there's no reason to let other versions influence our lock file.

dstufft · 2026-04-30T16:39:00Z

+    "Babel",
+    "bcrypt",
+    "boto3",
+    "celery[redis]>=5.6.2,<5.7",


Suggested change

"celery[redis]>=5.6.2,<5.7",

"celery[redis]>=5.6.2",

Currently we're not really using these pins, we're regularly updating the upper bounds (via dependabot) so we're tracking the latest version. With uv an upper bounds won't get automatically bumped liked that, because the lock file is what keeps you from upgrading.

dstufft · 2026-04-30T16:39:43Z

+    # https://github.com/jazzband/pip-tools/issues/1577
+    "kombu[redis]>=5.6,<5.7",


Suggested change

# https://github.com/jazzband/pip-tools/issues/1577

"kombu[redis]>=5.6,<5.7",

this was added due to a pip-tools bug that uv doesn't have so there's no reason to keep this.

dstufft · 2026-04-30T16:40:14Z

+    "rfc3986",
+    "sentry-sdk",
+    "pypi-attestations==0.0.29",
+    "sqlalchemy[asyncio]>=2.0,<3.0",


Suggested change

"sqlalchemy[asyncio]>=2.0,<3.0",

"sqlalchemy[asyncio]>=2.0",

Currently we're not really using these pins, we're regularly updating the upper bounds (via dependabot) so we're tracking the latest version. With uv an upper bounds won't get automatically bumped liked that, because the lock file is what keeps you from upgrading.

dstufft · 2026-04-30T16:45:52Z

+# requirements/*.txt files. The migration to pyproject.toml + uv.lock should
+# not silently bump versions; future upgrades happen via deliberate PRs that
+# remove a constraint and re-lock.
+constraint-dependencies = [


Once the uv.lock file has been added with these constraints, you can delete this from the pyproject.toml.

Creating the uv.lock with these constraints will ensure the lock file is generated with these constraints in mind, and then once we have that uv.lock, uv will ensure that we don't change versions without an explicit upgrade happening.

dstufft · 2026-04-30T16:46:53Z

+    "redis>=2.8.0,<7.0.0",
+    "rfc3986",
+    "sentry-sdk",
+    "pypi-attestations==0.0.29",


Suggested change

"pypi-attestations==0.0.29",

"pypi-attestations",

Currently we're not really using these pins, we're regularly updating the upper bounds (via dependabot) so we're tracking the latest version. With uv an upper bounds won't get automatically bumped liked that, because the lock file is what keeps you from upgrading.

dstufft · 2026-04-30T16:51:53Z

 	@exit 1

-.state/docker-build-base: Dockerfile package.json package-lock.json requirements/main.txt requirements/deploy.txt requirements/lint.txt requirements/tests.txt requirements/dev.txt
+.state/docker-build-base: Dockerfile package.json package-lock.json pyproject.toml uv.lock


We probably only need this to depend on uv.lock and not pyproject.toml, the uv.lock file depends on the relevant content in pyproject.toml, but there is also irrelevant content in pyproject.toml that would needlessly trigger a rebuild.

dstufft · 2026-04-30T16:53:15Z

+[project]
+name = "warehouse"
+version = "0.0.0"
+description = "PyPI's web application"


Suggested change

description = "PyPI's web application"

this is a nit, but this field isn't required and there's no real benefit to specifying it that I can tell.

dstufft · 2026-04-30T16:53:57Z

@@ -1,3 +1,480 @@
+[project]
+name = "warehouse"
+version = "0.0.0"


Suggested change

version = "0.0.0"

version = "1"

warehouse is stable 😉

dstufft · 2026-04-30T18:05:17Z

          path: |
              dev/.mypy_cache
-          key: ${{ runner.os }}-mypy-${{ env.pythonLocation }}-${{ hashFiles('requirements.txt', 'requirements/*.txt') }}
+          key: ${{ runner.os }}-mypy-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml', 'uv.lock') }}


Suggested change

key: ${{ runner.os }}-mypy-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml', 'uv.lock') }}

key: ${{ runner.os }}-mypy-${{ env.pythonLocation }}-${{ hashFiles('uv.lock') }}

The uv.lock is enough to know if the dependencies has changed and avoids unrelated changes to pyproject.toml invalidating the cache.

dstufft · 2026-04-30T18:14:38Z

You probably want to update the dependabot config to tell it to use uv, and the docker-compose.yml file to make sure that we're giving uv a cache location in .cache to persist with the other caches at runtime (used for make deps etc).

dstufft · 2026-04-30T18:19:11Z

 set -ex

+# Fail if pyproject.toml has changed without regenerating uv.lock.
+uv lock --check


This is really fast, so it's probably not a big deal either way, but given make deps still exists/works, that uv itself will generally enforce this, and CI has the Dependencies matrix item still, is this providing much value here?

I'm ok either way, just asking if it is :)

dstufft · 2026-04-30T18:20:28Z

 FROM python:${PYTHON_IMAGE_VERSION} AS docs

+# Pull the uv binary in so we can use it as a faster pip / pip-compile.
+COPY --from=ghcr.io/astral-sh/uv:0.11.7 /uv /uvx /usr/local/bin/


Can dependabot update this?

Either way, we reference the hardcoded version several times, it would be good to extract that into a variable like was done with PYTHON_IMAGE_VERSION.

notatallshaw · 2026-05-02T18:54:46Z

What version of pip was this benchmarked against?

I don't know where bottlenecks are in this CI but there were big improvements to metadata processing, resolver heuristics, and memory consumption in 26.1. I'm not saying to not go with uv, it has a lot of functionality pip never will, but my long term aim is performance parity.

JacobCoffee · 2026-05-02T19:37:18Z

it may have been 26.0, ill retry with latest

JacobCoffee added 5 commits April 27, 2026 18:18

poc move to uv

4e424ba

ci: add uv lock --check to bin/lint

b76e18f

Catches the case where someone edits pyproject.toml dependencies but forgets to regenerate uv.lock. Runs first in bin/lint so it fails fast before the longer ruff/mypy/djlint passes.

JacobCoffee force-pushed the poc/uv-localdev branch from 605eb90 to 78f3a53 Compare April 27, 2026 23:27

JacobCoffee and others added 2 commits April 27, 2026 18:36

JacobCoffee changed the title ~~(POC) Move to uv~~ (POC) Investigate uv Apr 27, 2026

JacobCoffee and others added 3 commits April 27, 2026 18:45

JacobCoffee mentioned this pull request Apr 28, 2026

(POC): Investigate webpack and frontend tooling alternatives #19954

Closed

dstufft reviewed Apr 30, 2026

View reviewed changes

LastWindBender mentioned this pull request Apr 30, 2026

Added dependencies for uv. #19794

Closed

		# https://github.com/jazzband/pip-tools/issues/1577
		"kombu[redis]>=5.6,<5.7",

	key: ${{ runner.os }}-mypy-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml', 'uv.lock') }}
	key: ${{ runner.os }}-mypy-${{ env.pythonLocation }}-${{ hashFiles('uv.lock') }}

Conversation

JacobCoffee commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JacobCoffee commented Apr 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dstufft commented Apr 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

notatallshaw commented May 2, 2026

Uh oh!

JacobCoffee commented May 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JacobCoffee commented Apr 27, 2026 •

edited

Loading