Skip to content

(Re)implement ipex functionality#23432

Draft
HeroCC wants to merge 2 commits into
pantsbuild:mainfrom
HeroCC:ipex-v2.1
Draft

(Re)implement ipex functionality#23432
HeroCC wants to merge 2 commits into
pantsbuild:mainfrom
HeroCC:ipex-v2.1

Conversation

@HeroCC

@HeroCC HeroCC commented Jun 16, 2026

Copy link
Copy Markdown

draft mode. Please don't merge yet - I have some fixes from our plugin to backport to this.

Problem

We are running into the same problem as pex-tool/pex#789. Ie., we have massive pex artifacts (in this case, pyspark) which are slow to upload, and expensive to store (we don't need 100s of copies of embedded pyspark stored at the same time).

Solution

(Re)introducing ipex - a pex file packaged with sources + a dependencies lockfile. When run, it rehydrates dependencies from whichever requirements file was requested.

Essentially the same as #8940

Additional Notes

Ths code here is almost entirely codex w/ GPT-5.5 + some manual tweaks by me. I told it to read the changes from #8940 and reimplement it on top of modern pants, then prompted it to add some more unit test cases we care about (full lockfile resoluton).

@jsirois

jsirois commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

@HeroCC --layout packed is new since the time of the issue & PR you reference. Is that not enough? (You may also want --no-pre-install-wheels) That layout is explicitly designed to be cache friendly / rsync friendly and Pants itself has long since switched to using it internally.

If it's not enough, you should at least detail why to prove you've done your homework for future readers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants