Cache Go module downloads in a persistent named cache (GOMODCACHE)

[rdeknijf]

Disclaimer: Like my previous Go PR, I'm still not primarily a golang developer. However, the fact that Golang doesn't work properly in Pants has been the bane of my existence for like 2 years now. The moment that Mythos/Fable dropped I immediately had it dig deeply into whether the whole road to proper-golang-in-pants could be opened up. So this is all by Claude Code with Fable 5 (xhigh), with consults to GPT 5.5 (xhigh) and Gemini 3.1 Pro. I've had it check and recheck, I ran many different roles over it, and I had it explain and re-explain it to me, and then I checked myself. The cache-partitioning commit that fixes the CI failure came later and was written by Claude Opus 5, then reviewed by three separate Fable passes (semantics, security, tests) before I pushed it; same drill of checking and re-checking.

So, as much as I dislike AI slop and am worried about AI PR overload, I've done my very best to avoid exactly that while still using AI. I hope we can get this road unblocked.

The rest is Fable talking:

---

Every Go sandbox currently gets a fresh GOMODCACHE, so a cold build re-downloads every third-party module from the network, and so does any change that invalidates the cached download processes. This is #13390; I posted the design there in #13390 (comment) and benjyw signed off on it, deferring the Go specifics to tdyas.

This PR gives the two download processes in third_party_pkg.py (the only Go processes that run with allow_downloads=True) a go_mod_cache named cache, used purely as a download accelerator. Captured digests remain the source of truth: modules are copied out of the shared cache into the sandbox's gopath/pkg/mod before capture, so process results are byte-identical whether the cache is warm or cold, and compile, link, and vet sandboxes never see the shared cache at all. They keep running with GOPROXY=off against captured digests, exactly as today.

Mechanics:

• Download and copy happen in one process: a fetch mode in __run_go.sh runs go mod download -json, emits the metadata on stdout, and copies the extracted module and its go.mod out of the cache into gopath/pkg/mod. Keeping it one process preserves self-healing: wipe the named cache and the next run of the same process repopulates it.

• GOMODCACHE points inside __gomodcache, a sibling of the captured gopath/ tree, so output_directories=("gopath",) can never capture the shared cache by accident.

• The cache is partitioned by the checksums the caller expects. Go stores a module at <module>@<version>, a path that says nothing about the bytes, so one shared cache would serve whatever the first build stored there to every later build. A disagreement then means a checksum-mismatch SECURITY ERROR that survives until someone wipes the cache by hand, and re-tagging a version (which private modules and internal proxies do) is enough to trigger it. The fetch process therefore uses a partition derived from <module>@<version> plus the go.sum entries the caller expects for it, and module-graph analysis uses one derived from the go.mod/go.sum pair in its input digest. Stable checksums mean a stable partition, so builds and repos that agree still share their downloads. Partitions accumulate, as named caches are never garbage collected today, but they stay small: an analysis partition holds only .mod text files, and a fetch partition holds the same extracted modules a single shared cache would have held, duplicated only when the recorded checksums for a module actually change. Each partition is a self-contained directory, so pruning by mtime later needs no bookkeeping.

  The partition also covers the environment that decides where a download comes from and whether it gets checked (GOPROXY, GOPRIVATE, GOSUMDB and the rest), which matters for modules with no go.sum entries: those have no checksums to partition by, so builds configuring a different proxy or checksum database must not share downloads.

• -modcacherw is added via GOFLAGS so the cache can be pruned with plain rm -rf.

• Checksum verification is unchanged: downloads are still verified against the synthetic per-module go.sum introduced in backend/go: deduplicate third-party module downloads across go.mods #23261, and nothing touches GONOSUMCHECK or GOFLAGS beyond -modcacherw. The trust model of the shared cache is the same as the default ~/go/pkg/mod on a developer machine.

• Remote execution degrades gracefully: without --remote-execution-append-only-caches-base-path the cache mount is absent, the fetch falls back to a sandbox-local directory, and behavior is identical to today.

• ModuleDescriptors.go_mods_digest is removed; it had no consumers.

On a 3-go.mod / 206-module reproducer with a cold engine store, a warm module cache eliminates all 103 network downloads (the slowest module went from 8.6 s of network fetch to 3.1 s of local copy) and module-graph analysis (go list -m) stops hitting the network entirely. Cold wall clock on that reproducer is unchanged within noise, because compile time dominates there and proxy.golang.org is fast from my machine. The practical win is cold CI builds and slow or rate-limited proxies, where today every cold build pays the full download set again.

Tests: warm-vs-cold digest equality (which also catches absolute paths leaking into captured digests), named cache populated and writable after a download, module analysis served from the cache, and two builds publishing the same module@version with different contents through a local proxy, which is the collision the partitioning prevents. Release note in docs/notes/2.34.x.md, including the one-time invalidation from the __run_go.sh change.

Note: this touches the same __run_go.sh heredoc as #23420 (GOTOOLCHAIN pin); whichever lands second needs a trivial rebase.

[rdeknijf]

Confirming the description's framing with a real-repo measurement. On a 24-module monorepo A/B (warm daemon and store, remote cache off), cold wall time is flat, exactly as the description says: on a fast connection, downloading was never the slow part. This PR only touches the two download processes, so it does not affect compile, link, or the warm incremental build at all. The win remains cold CI and slow or rate-limited proxies, where a cold build otherwise pays the full download set every time. Methodology and the per-lever breakdown are on #20274.

[tdyas]

Are the benchmarking script available somewhere?

[tdyas]

cc @jgranstrom

[jgranstrom]

This looks generally good to me. I benchmarked it as well on a repo with ~500 modules. Correctness looks good. Cache behavior also checks out, only affected modules need to redownload to the shared cache, then are captured via action cache. And with the slimmer capture it reduces the store outside the named cache. 👍

The one regression, or rather trade-off, I found is on a completely cold download of an entire go.mod (no named cache, no action cache). In my benchmark this regressed about 20-25%. This looks to be the additional materialization-pass from the shared cache to the sandbox's gopath for capture. Since they are already extracted in the named cache, this can be a lot of files, ~65k in my case, to materialize. However, I think it might still be the right trade-off here given the multiple levels of caching mitigating it.

A follow-up to address that regression could be to extract directly into the sandbox (as before) and keep only the downloaded archives in the named cache, though I'm not sure there's a trivial path to that.

I can rebase my #23499 on this once it's landed.

[tdyas]

I added a few comments about not hard-coding binary paths for CLI tools.

Relatedly, this change increases the complexity of the shell script injected for the Process generated from a GoSdkProcess. What are your thoughts on adding assertions that the exact expected artifacts were in fact copied into the new cache?

[tdyas]

This code should not assume that the environment has cat at /bin (nor any other utility for that matter). The rule code should use BinaryPathRequest (or one of its tool-specific subclasses) to request the engine to find the actual path of the tool to be used.

In this case, the rule should request CatBinary from pants.core.util_rules.system_binaries.

[rdeknijf]

Done, it now requests CatBinary and interpolates the discovered path. Same treatment for the other utilities in this script: CpBinary, MkdirBinary, and PwdBinary for the pwd call further up, which was the remaining hard-coded /bin path. The pwd one passes -P explicitly, since go needs the resolved physical path and a bare pwd would return a logical one containing symlinks. That was implicit in the old hard-coded call, so it seemed better to state it than to depend on which pwd gets found.

All the paths are shlex.quoted before going into the script.

[tdyas]

Similar comment to the one for cat: Use the path from MkdirBinary.

[rdeknijf]

Done, this uses MkdirBinary now. I converted the pre-existing /bin/mkdir on the GOPATH/GOCACHE setup line in the same pass, since it had the same problem.

[tdyas]

You can ask the engine to find a grep binary if you want. Trade off is requiring grep to be present versus complicated bash logic.

[rdeknijf]

I went with keeping the bash logic. The trade-off you describe is real, and I do not think it is clear-cut, but requiring grep in every Go sandbox to parse two fields felt like the larger cost.

I did tighten the parse so it is not silently wrong: it now rejects missing or unrecognized fields rather than slicing whatever it got. It is still string matching, so I will not claim it cannot misparse a sufficiently strange path. Happy to switch to GrepBinary if you would rather take the dependency, it is a small change.

[tdyas]

Use the paths from the applicable BinaryPathRequest result types in pants.core.util_rules.system_binaries.

[rdeknijf]

Done. go_sdk_invoke_setup now takes CatBinary, CpBinary, MkdirBinary and PwdBinary as rule parameters and interpolates the resolved paths, so there are no hard-coded /bin paths left in this script.

[cburroughs]

Thanks for the contribution. We've just branched for 2.33.x, so merging this pull request now will come out in 2.34.x, please move the release notes updates to docs/notes/2.34.x.md if that's appropriate.

[rdeknijf]

@tdyas I answered your four review comments inline on the threads themselves, so this is only the part that does not belong on a specific line.

On the assertions you asked about: the script now checks after the copy that the destination .mod file and module directory exist. The Python rule rejects metadata paths that do not contain the expected module-cache marker instead of silently slicing them, and fails if the captured snapshot has no files under the expected module path. The tests assert the exact artifacts land in both the named cache (extracted sources, the download cache .mod entry, and an owner-writable module dir, which proves -modcacherw reached the process) and the captured digest (uuid.go, go.mod, and cache/download/.../@v/v1.3.0.mod).

On your question about the benchmarking scripts: https://gist.github.com/rdeknijf/ade1f4f652e234c01721670097c72f0d. measure-coldwarm.sh is the one behind the numbers in this PR, running a command cold and then warm against the same stores with a fresh PANTS_LOCAL_STORE_DIR and PANTS_NAMED_CACHES_DIR each time. It is a personal rig rather than a supported tool, and Linux only since it reads /proc. Two things in the README that cost me time: taskset does not actually cap Pants to N cores, because sandboxed children get reset affinity, so you need a cgroup over the whole tree; and child CPU totals have to come from the launcher's cutime/cstime rather than sampling, which misses most of the short-lived compiles. Happy to fold any of it into build-support/ if you would find it useful in-tree.

@jgranstrom thanks for benchmarking this on a 500-module repo, that is the validation it needed most. Agreed on the trade-off, with one clarification on where the cost sits: the copy runs on every fetch execution, so an action cache hit is what skips it. What is specific to the cold case is the extra pass, since with a cold module cache you pay the extraction into it and then the copy out of it, where before there was only the extraction. Once the module cache is warm the copy replaces work the old code did anyway.

Your follow-up direction looks right to me. Extraction looks deterministic enough for it, since Go writes 0444 regular files and does not carry archive timestamps through, and Pants digests only record content, path and the executable bit. The checksumming does less work for us than it first appears though: Go verifies an archive when it first downloads it, but once the .zip and .ziphash are in the cache it trusts the stored hash, so a corrupted persistent archive is not guaranteed to be caught later. The other open question is the mount point. It is at __gomodcache today, outside the captured gopath, and mounting it under gopath/pkg/mod/cache/download would put named cache symlinks in the path of output capture, so that wants a cold/warm digest test before anyone commits to the design. I will file an issue once this lands so it does not get lost.

On #23499, that ended up landing ahead of this one, so there is nothing to rebase on your side. I rebased this PR onto current main on the 24th, so your change is already in here.

@cburroughs release note moved to docs/notes/2.34.x.md, and this is rebased on current main.

[jgranstrom]

The test failures look to be because we re-use a test fixture go-sample-for-test@v0.0.1 across tests with different contents, so the hash verification fails for whoever lost the race to the named cache. The fix could be either naming them uniquely per test/variation, or use an isolated named cache per test.

Worth noting that this also fixes another issue that I noticed but not sure if it's been mentioned. Before, since we downloaded without -modcacherw, every sandbox for a go module download was silently partially leaked. Sandbox cleanup couldn't delete the extracted module cache inside the sandbox (gopath/pkg/mod) because of the read-only directory permissions, so extracted go mods leaked on every download. Since this PR adds -modcacherw, those sandboxes can now be fully torn down at cleanup.

[rdeknijf]

@tdyas @jgranstrom Pushed a fix for the CI failures on this branch, and moved the release note to docs/notes/2.34.x.md as @cburroughs asked.

What went wrong: the named cache was a single shared GOMODCACHE. Go stores an extracted module at <module>@<version>, a path that carries nothing about the bytes, so the first build to store something there decides what every later build receives. The Go backend's own tests publish pantsbuild.org/go-sample-for-test@v0.0.1 with different file contents per test through a local module proxy, so once two of them ran in the same shard the second one failed checksum verification with Go's SECURITY ERROR. It surfaced after the rebase past #23499, which changed how many of those per-test modules a single shard downloads.

The same hazard exists outside the test suite: re-tagging a version, which private modules and internal proxies do, leaves a stale extraction behind that fails every later build until someone wipes the cache by hand.

The fix partitions the named cache. The module fetch uses a partition derived from <module>@<version> plus the go.sum entries the caller expects for it, and the module-graph analysis uses one derived from the go.mod/go.sum pair in its input digest. Builds that disagree about a module's contents land in different partitions, so a stale entry cannot poison a later build, and changing the recorded checksums re-downloads instead of failing. Builds that agree still share the download, which is the normal case. A module with no recorded checksums has nothing to partition by, and there the integrity of a download rests on the environment instead: which proxy served it and whether the checksum database was consulted. The partition therefore also covers GOPROXY, GOPRIVATE, GOSUMDB and the rest of that set.

Two tests cover it: one publishes the same module@version twice with different contents through a local proxy and asserts both builds get their own sources, and one runs the same module twice with the proxy deleted in between, which only succeeds if the second run was served from the partition the first run populated.

[benjyw]

OP and reviewers - I'm just checking in on the status of this. What needs to happen to merge it? Thanks.

[jgranstrom]

I would have liked to see more benchmarks on it. I ran a slightly more comprehensive benchmark on it against main on ~700 modules and I'm considering if this needs the additional pass fix before landing.

On a not-slow network I could not find a configuration where this is a win. Not cold cache, not warm cache. Even with the named cache fully populated it is significantly slower than having no cache at all, and it costs 80% more disk across the named cache and local store.

I wanted to attribute the regression properly, so I did a quick experiment using GOPROXY so that there is no additional materialization while still adding the named cache. I.e. Go downloads the archive, we publish it to the named cache, and later builds reach for it via GOPROXY=file:// instead of copying an extracted tree back into the sandbox. That came in at ~2% wall time and ~2% disk against the same baseline.

To be clear though, that prototype is not a speedup either. Its warm-cache cell is +2.9% against the baseline and about 8s slower than its own cold run. Reading an archive back off local disk and copying it into the sandbox costs about what downloading it costs here, so the named cache does not buy wall time under either transport on my benchmarks. The difference is only in what having the cache costs you: ~2% instead of ~22%.

Another secondary cost introduced is the sandbox cleanup. That was silently skipped on main because Go leaves module directories read-only, so remove_dir_all fails and the error is swallowed. Main leaks ~700 sandboxes / 2.1GB per cold run, indefinitely. -modcacherw correctly fixes that and adds the cost back, which is significant and comes with contention problems. That is a separate issue in itself, but for the benchmarks to be fair I measured against main with that flag enabled so cleanup runs on both sides.

Isolating with --keep-sandboxes splits the 39.4s gap to that baseline into +23.5s of actual work (the copy) and +15.9s of cleanup, so roughly 60/40. The cleanup half is not independent: this PR does more work, cleanup runs concurrently with it, and so cleanup gets slower too. Any improvement to the sandbox cleanup would therefore shrink this PR's regression as well as the baseline's.

One thing worth reconsidering is the motivating case. #13390 is about a go.mod change forcing a full re-download. That is already handled on main: each download is keyed on (name, version, minimum_go_version, build_opts, go_sum_entries) with cross-go.mod dedup, so changing one dependency leaves the rest as store hits. What remains is store-cold-but-cache-warm, which is the +21.6% row below, measured against a baseline that simply downloads. The cache only pays for itself once download time dominates, which on ~490MB of archives means significantly slower network than I'm benching on.

And note that named caches are excluded from the action cache key, so with a warm local store these processes never execute and the cache is never read at all.

Numbers below. Base 23be51b, free-threaded CPython 3.14.6, --no-pantsd, no remote cache, fresh store and named-cache dir per run. Variants were run alternating rather than in blocks so machine drift affects them equally, each run verified it was actually running the variant it claimed, and runs producing the wrong target count were discarded. Process counts identical across every arm (1,549). Measured under both third_party_target_granularity settings; every ratio agreed within ~2 points, so the numbers are granularity-independent.

Fetch and analysis (pants list, cold store)

cell	n	runs	mean	vs main	vs leak-free baseline
cleanup disabled (--keep-sandboxes, control)	2	91.6, 95.0	93.3s	-42.4%	-49.0%
main	2	162.4, 161.8	162.1s	-	-11.3%
main + -modcacherw (leak-free baseline)	10	177.9-187.8	182.8s	+12.8%	-
GOPROXY prototype, cold cache	2	180.2, 180.1	180.1s	+11.1%	-1.5%
GOPROXY prototype, warm cache	2	186.3, 189.9	188.1s	+16.0%	+2.9%
this PR, warm cache	2	216.3, 228.1	222.2s	+37.1%	+21.6%
this PR, cold cache	3	238.8, 235.8, 236.7	237.1s	+46.3%	+29.7%

Work vs cleanup, isolated with --keep-sandboxes

arm	total	work only	cleanup
main	162.1s	90.0s	72.1s
main + -modcacherw	182.8s	93.3s	89.5s
this PR, warm	222.2s	116.8s	105.4s

Disk (local store + named cache)

cell	store	named cache	total	vs main
main	2,771,028 KB	0	2,771,028 KB	-
this PR	2,329,344 KB	2,647,400 KB	4,976,744 KB	+79.6%
GOPROXY prototype	2,332,520 KB	503,924 KB	2,836,444 KB	+2.4%

In-process time and other scopes

measure	main	leak-free baseline	this PR	GOPROXY
total in-process time (warm)	1,079,523 ms	1,268,911 ms	1,616,836 ms	1,206,715 ms
hot path, warm store (5 runs, first 2 dropped)	4.06s	-	4.10s	4.07s
binary sha256	e1412da9...	-	e1412da9...	e1412da9...

The prototype only swaps transport on the fetch path. Metadata steps (go list -m -json all) download ~10MB of .mod files and capture nothing, so they keep GOMODCACHE. Cache reuse for the prototype is verified with GOPROXY="file://<cache>,off", i.e. no network fallback at all, which still produced the full target set and added zero bytes to the cache.

My recommendation is to look into the near-zero-cost wiring via GOPROXY before landing. The mechanism here is sound and the leak fix is valuable on its own, but even with a corrected baseline, extract-then-copy costs more than the network it saves in every cell I could measure and nearly doubles Go disk usage. Sharing only the archive would not make builds faster here either, but it makes the cost of having the cache available ~2% rather than ~22%, which matters on the slow networks where it might pay off.

[rdeknijf]

@benjyw @jgranstrom @tdyas Closing this one, and splitting out the part of it that is worth keeping.

Thanks in particular to @jgranstrom for the ~700-module benchmark. It changed my mind on the cache, and I would rather act on the data than defend the PR. The short version of why this should not land as-is:

• The motivating issue Hook up GOPATH to named_caches? #13390 is the go.mod-edit case, and that is already handled on main. Downloads are keyed on (name, version, minimum_go_version, build_opts, go_sum_entries) with cross-go.mod dedup (backend/go: deduplicate third-party module downloads across go.mods #23261), so editing one dependency leaves the rest as local-store hits. There is no full re-download to save.
• What remains is store-cold-but-cache-warm, and the numbers show that is a net loss on the benchmarked network: extract-then-copy costs more wall time than the download it replaces in every cell measured, and it adds about 80% to Go disk usage. Named caches are also excluded from the action-cache key, so a warm local store never runs these processes and never reads the cache at all.
• The cache only starts to pay off once download time dominates, i.e. on a much slower or rate-limited network than any measured workload here. That is a real scenario but not one I can benchmark a win on today.

The one piece of this PR that stands on its own is the sandbox leak fix. Without -modcacherw, Go leaves extracted module directories read-only, sandbox cleanup fails to remove them, and the error is swallowed, so every cold download leaks its sandbox (jgranstrom measured ~2.1 GB per cold run). I have pulled that out into a small standalone PR: #23636. It is a correctness fix, not a performance bet, and jgranstrom's "leak-free baseline" arm already measured its runtime effect.

If the slow-network case ever gets a concrete workload behind it, the right shape is the GOPROXY=file:// transport jgranstrom prototyped (share the unextracted download artifacts, let Go re-verify against the consumer go.sum) rather than the extract-then-copy here. I have written that up in the issue thread so it is not lost.