Cache Go module downloads in a persistent named cache (GOMODCACHE)

[rdeknijf]

Disclaimer: Like my previous Go PR, I'm still not primarily a golang developer. However, the fact that Golang doesn't work properly in Pants has been the bane of my existence for like 2 years now. The moment that Mythos/Fable dropped I immediately had it dig deeply into whether the whole road to proper-golang-in-pants could be opened up. So this is all by Claude Code with Fable 5 (xhigh), with consults to GPT 5.5 (xhigh) and Gemini 3.1 Pro. I've had it check and recheck, I ran many different roles over it, and I had it explain and re-explain it to me, and then I checked myself.

So, as much as I dislike AI slop and am worried about AI PR overload, I've done my very best to avoid exactly that while still using AI. I hope we can get this road unblocked.

The rest is Fable talking:

---

Every Go sandbox currently gets a fresh GOMODCACHE, so a cold build re-downloads every third-party module from the network, and so does any change that invalidates the cached download processes. This is #13390; I posted the design there in #13390 (comment) and benjyw signed off on it, deferring the Go specifics to tdyas.

This PR gives the two download processes in third_party_pkg.py (the only Go processes that run with allow_downloads=True) a go_mod_cache named cache, used purely as a download accelerator. Captured digests remain the source of truth: modules are copied out of the shared cache into the sandbox's gopath/pkg/mod before capture, so process results are byte-identical whether the cache is warm or cold, and compile, link, and vet sandboxes never see the shared cache at all. They keep running with GOPROXY=off against captured digests, exactly as today.

Mechanics:

• Download and copy happen in one process: a fetch mode in __run_go.sh runs go mod download -json, emits the metadata on stdout, and copies the extracted module and its go.mod out of the cache into gopath/pkg/mod. Keeping it one process preserves self-healing: wipe the named cache and the next run of the same process repopulates it.
• GOMODCACHE points at __gomodcache, a sibling of the captured gopath/ tree, so output_directories=("gopath",) can never capture the shared cache by accident.
• -modcacherw is added via GOFLAGS so the cache can be pruned with plain rm -rf.
• Checksum verification is unchanged: downloads are still verified against the synthetic per-module go.sum introduced in backend/go: deduplicate third-party module downloads across go.mods #23261, and nothing touches GONOSUMCHECK or GOFLAGS beyond -modcacherw. The trust model of the shared cache is the same as the default ~/go/pkg/mod on a developer machine.
• Remote execution degrades gracefully: without --remote-execution-append-only-caches-base-path the cache mount is absent, the fetch falls back to a sandbox-local directory, and behavior is identical to today.
• ModuleDescriptors.go_mods_digest is removed; it had no consumers.

On a 3-go.mod / 206-module reproducer with a cold engine store, a warm module cache eliminates all 103 network downloads (the slowest module went from 8.6 s of network fetch to 3.1 s of local copy) and module-graph analysis (go list -m) stops hitting the network entirely. Cold wall clock on that reproducer is unchanged within noise, because compile time dominates there and proxy.golang.org is fast from my machine. The practical win is cold CI builds and slow or rate-limited proxies, where today every cold build pays the full download set again.

Tests: warm-vs-cold digest equality (which also catches absolute paths leaking into captured digests), named cache populated and writable after a download, and module analysis served from the cache. Release note in docs/notes/2.33.x.md, including the one-time invalidation from the __run_go.sh change.

Note: this touches the same __run_go.sh heredoc as #23420 (GOTOOLCHAIN pin); whichever lands second needs a trivial rebase.

[rdeknijf]

Confirming the description's framing with a real-repo measurement. On a 24-module monorepo A/B (warm daemon and store, remote cache off), cold wall time is flat, exactly as the description says: on a fast connection, downloading was never the slow part. This PR only touches the two download processes, so it does not affect compile, link, or the warm incremental build at all. The win remains cold CI and slow or rate-limited proxies, where a cold build otherwise pays the full download set every time. Methodology and the per-lever breakdown are on #20274.