Run pants under free threaded python

[tobni]

Switches Pants' own runtime to free-threaded CPython and makes the engine and
Pants' Python code thread-safe without the GIL.

This has been validated as running without error in my companies CI, and nets great speedups.

Most important callouts:

• Keep a persistent per-thread PyThreadState in the executor. Without
  this, every Python::attach cycle tears down the thread state, and on 3.14t
  that abandons the thread's mimalloc heaps. Created per tokio thread, torn down in on_thread_stop (blocking
  threads are recycled, so the state must not leak per thread creation).
• Makes run-scoped internals thread-safe: rule_visitor, exception_sink, logging, build_root, run_tracker.

I used #23437 as a launching point.

I have on speculation set this as shipping in 2.34.x, as I believe 2.33 is too far along the dev-cycles. But I also do not see any reason to delay further.

Blocks pantsbuild/scie-pants#541

Edit:
This branch now serves as proof of green CI of the accumulation of changes, and will be rebased as its parts are accepted incrementally:

• Free-threaded: Upgrade psutil #23487
• Free-threaded: Export free threaded site packages #23488
• Free-threaded: Engine groundwork for performant free-threading #23489
• Free-threaded: Drop hdrhistorgram python dependency #23490
• Free-threaded: Thread safe internals #23491
• Free-threaded: Native, thread-safe instance memoization #23492

This PR will house the changes to actually enable a free-threaded pants release, coupling it to pantsbuild/scie-pants#541.

[cburroughs]

(Some adjacent or related issues #22790 #22946 #15840 #21518 so I don't need to search for them again.)

@tobni As I understand it there isn't a stable ABI for free threaded until 3.15. Having dug in to do this work, do you have a sense of how much of a problem or risk that is (or is not)?

[tobni]

@cburroughs Risk is close to none afaik, it is mostly a pyo3 compat (meaning we couldnt build it) issue. If we are allowed to assume pants is ran as a scie we defined, the internal 3rd party lockfile does the rest. I think that is ok, building a pants dist and targeting a different python would be highly special usecase.

If a plugin uses native deps in rule code that could bite them iff there is no 3.14t wheels for them.

[sureshjoshi]

Pretty substantial (and consequential) change - so I tacked on a bunch of reviewers

[tobni]

Once #23482 is in this will be rebased and the pex binary will again be stripped.

Edit: Done.

[tdyas]

General review comments:

1. This PR should probably be split into distinct components to make review easier. For example, the LockMap change and the related memoization changes can be a standalone PR.

2. Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

[tobni]

Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

We could vendor release versions with t-suffixes to make it target free-threaded? I'm doubtful of the value to be honest, but that'd be simple enough implementation wise (but requires CI/CD work).

I'd prefer opt-out though. I dont want to treat this as a scary change when it isnt, but I will also not claim 0 impact across all plugin code.

For the first comment: Yes, easily split. Each commit ~stands on its own already so I can do that branch-ectomy with ~0 effort.

[tdyas]

We could vendor release versions with t-suffixes to make it target free-threaded? I'm doubtful of the value to be honest, but that'd be simple enough implementation wise (but requires CI/CD work).

So like everything in software engineering, my view is "it depends." My preference here is that we not introduce a one way version gate on users when there is no one dedicated to fixing issues for them in a timely manner. I'd rather avoid users being potentially stuck on 2.32 or lower versions with no way to consume all of the other fixes Pants has in 2.33 unrelated to free-threading.

So how much more complex would it be to release a free-threading version of Pants alongside the GIL version? The main artifact is the engine's shared library (native_engine.so), can we just build it twice? (once with 3.14 GIL and once with 3.14 free-threaded)?

My view really depends on just how much additional work the CI/CD & release changes would be.

I'd prefer opt-out though. I dont want to treat this as a scary change when it isnt, but I will also not claim 0 impact across all plugin code.

Opt-out is fine. The concern for me is having the ability to control adoption of the change with some sort of flag versus forcing users to stay on a prior versions because there is no escape hatch from this change. I submit that Incremental adoption with an escape hatch makes for a better user experience.

[tobni]

So like everything in software engineering, my view is "it depends." My preference here is that we not introduce a one way version gate on users when there is no one dedicated to fixing issues for them in a timely manner. I'd rather avoid users being potentially stuck on 2.32 or lower versions with no way to consume all of the other fixes Pants has in 2.33 unrelated to free-threading.

You used specific version numbers so I'll just clarify that I do not think this is a 2.33.x change but rather a 2.34.x feature. The point is of course still valid. I'm not sure I agree that this is a strong case for pants and free threading though, we have a patch-version series for fixes and we vendor a binary. We can choose to only release free threading and address eventual bugs. Because it will be bugs, users being "stuck" will be on some flaky CI-runs surfacing races we missed, most likely.

So how much more complex would it be to release a free-threading version of Pants alongside the GIL version? The main artifact is the engine's shared library (native_engine.so), can we just build it twice? (once with 3.14 GIL and once with 3.14 free-threaded)?

I dont know exactly, the main effort is figuring that out, and is the counterweight in terms of value I was refering to.

[tobni]

Now I know some more - It'd have to be a separate pants.toml field to not mess with PEP440 compat, but otherwise it goes:

1. One release, two asset sets. The GitHub release has to publish both scies per platform eg: pants.2.34.0-linux-x86_64.scie and a free threaded sibling whose naming is TBD.
2. Selection is a scie-pants config knob. scie-pants reads from pants.toml. scie-pants maps the pants.toml field to the asset-name suffix and fetches.

Essentially a question of if we're okay to publish 2x scies.

[tobni]

General review comments:

1. This PR should probably be split into distinct components to make review easier. For example, the LockMap change and the related memoization changes can be a standalone PR.
2. Is it feasible to make the free-threaded Python be a runtime choice? I'd rather we prefer it be an opt-in experiment initially. If it can't be done based on a runtime option, maybe let scie-pants know how to configure it and download a separate Pants artifact with free-threaded support?

@tdyas 1 is now done. See the list in the pull request description.