Skip to content

Commit 3c84c8e

Browse files
bodonoclaude
bodono
and
claude
authored
Add free-threading (no-GIL) support for Python 3.13t+ (#175)
* Add free-threading (no-GIL) support for Python 3.13t+ Enables scs to run with the GIL disabled in free-threaded CPython builds. C extension changes: - Add PyMutex per-instance lock (only in Py_GIL_DISABLED builds) to serialize concurrent access to the SCS workspace from multiple threads - Lock around scs_solve, scs_update, and scs_finish (pure C calls safe to hold across Py_BEGIN_ALLOW_THREADS) - No locking in SCS_init (object is thread-local during construction) - Declare module as GIL-not-used via PyUnstable_Module_SetGIL Build/packaging: - Enable cpython-freethreading in cibuildwheel to ship 3.13t+ wheels - Add free-threading PyPI classifier CI: - Add free-threading test workflow (3.13t + 3.14t) alongside existing build.yml Tests: - 15 new concurrency tests covering: independent instances, shared instances, solve+update sequences, warm starts, legacy API, stress tests, result isolation Closes #130 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix free-threading CI: create venv before installing packages uv pip install requires a virtual environment or --system flag. Use uv venv to create a proper venv for the free-threaded Python. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix free-threading CI: install build deps for --no-build-isolation meson-python and meson must be in the venv when using --no-build-isolation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix free-threading CI: use build isolation instead of --no-build-isolation Let the build system handle its own dependencies (meson-python, numpy) via pyproject.toml build-system.requires, rather than manually installing them and using --no-build-isolation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix free-threading CI: install pytest-timeout for --timeout flag Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address free-threading review feedback and make all tests thread-safe Correctness fixes from @ngoldbaum's review: - PyDict_GetItemString -> PyDict_GetItemStringRef (strong refs) - PyList_GetItem -> PyList_GetItemRef (strong refs) - Proper Py_DECREF and scs_free on all error paths - Move self->work NULL check under lock (TOCTOU fix) in SCS_solve/SCS_update - Check PyThread_acquire_lock return value (PY_LOCK_ACQUIRED) - Add comment to SCS_finish explaining unchecked lock acquire in dealloc path - Fix SCS_update lock release ordering with comment explaining why it differs from SCS_solve (avoid holding instance lock while waiting for GIL) Thread-safe RNG in all test files: - Replace all np.random.seed() + global RNG with local RandomState instances - Each test file gets unique seeds so tests exercise different random data - Backend-variant tests (direct/dense/mkl/cudss) share seeds intentionally since they test the same problem with different linear system solvers Testing infrastructure: - 27 new threading tests covering borrowed refs, TOCTOU races, concurrent solve+update, re-init races, error path lock release, stress tests - GIL re-enable detection in conftest.py (modeled after NumPy) - pytest-run-parallel in CI (--parallel-threads=4 --iterations=3) - TSan CI job with cached CPython 3.14t build - faulthandler_timeout and thread_unsafe markers in pyproject.toml - tsan-suppressions.txt for known CPython-internal races Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix thread-safe signal handling and link pthreads Update scs_source submodule to thread-safe-ctrlc branch which adds mutex-protected reference counting for signal handler registration, fixing TSan-detected data races when multiple threads solve concurrently. Add pthreads dependency to meson.build for the ctrlc mutex. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Set OPENBLAS_NUM_THREADS=1 for TSan CI OpenBLAS spawns internal threads that are not instrumented with TSan annotations, causing false positive data race reports (e.g. in dsyrk worker threads racing with the caller after the BLAS call returns). Forcing single-threaded BLAS under TSan is standard practice. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add ASan CI job via sanitizer matrix Refactor the TSan CI job into a matrix over {tsan, asan}. Both build CPython 3.14t from source with the corresponding sanitizer flag. Runtime options (TSAN_OPTIONS, ASAN_OPTIONS, OPENBLAS_NUM_THREADS) are set via GITHUB_ENV in a shared setup step. ASan uses detect_leaks=0 to avoid false positives from CPython's internal memory allocator. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Set sanitizer env vars before build step ASan's LeakSanitizer was killing the build because ASAN_OPTIONS was only set after the build. Move the env setup step before pip install so detect_leaks=0 applies during compilation too. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Move sanitizer suppressions to test/, add LSan suppressions Move tsan-suppressions.txt to test/ and add lsan-suppressions.txt for CPython-internal leaks. Use LSan suppressions instead of detect_leaks=0 so that real SCS leaks are still caught. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Stabilize Accelerate backend tests * Keep Accelerate random-problem tests deterministic * Add faulthandler_exit_on_timeout and simplify CI pytest flags Set faulthandler_exit_on_timeout = true in pyproject.toml so hanging tests actually terminate in CI instead of just dumping tracebacks. Remove redundant -p no:faulthandler -o faulthandler_timeout=600 flags from the free-threading CI since the config is now in pyproject.toml. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Address review: update FT classifier, drop cibuildwheel enable, cleanup - Fix Free Threading trove classifier: drop extraneous "Implementation ::" prefix, bump "1 - Unstable" to "3 - Stable" per PEP 779 and reviewer feedback (free-threading is no longer experimental in 3.14). - Drop `enable = ["cpython-freethreading"]`: deprecated in cibuildwheel 3.4.1, 3.14t wheels build without it. Side effect: we no longer ship 3.13t wheels, which aligns with upstream's migration push to 3.14t. - Drop `thread_unsafe_fixtures = ["capsys", "capfd"]`: pytest-run-parallel marks these automatically. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Drop re-init concurrency test, document construction as thread-local Per review discussion: the `SCS_init` guard against concurrent re-init of a live instance is not standard practice for CPython extensions — NumPy, SciPy, and similar libraries assume object construction is thread-local and do not lock `__init__`. The `test_reinit_while_solving` test also swallowed exceptions broadly, which hid rather than detected any real race on the `self->work` field. Remove the test and add a docstring note to `SCS.__init__` making the thread-local-construction assumption explicit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Use LinearSolver enum in test_concurrent_direct_and_indirect Missed during the rebase onto master: this test still passed the old `use_indirect=` boolean kwarg, which was removed in #189 in favor of the `linear_solver=scs.LinearSolver.*` enum. This broke the full CI matrix (wheel builds, accelerate builds, free-threading tests, sanitizers) on a single shared failure. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Mark test_resolve_auto_* as thread_unsafe These tests patch scs.sys and scs._load_module — module-level state that leaks to other tests running in parallel threads under pytest-run-parallel, causing MagicMock to be substituted for the real scs.SCS class in unrelated tests. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent f6b7dc6 commit 3c84c8e

23 files changed

Lines changed: 1485 additions & 171 deletions

‎.github/workflows/freethreading_tests.yml‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
name: Free-threading tests
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches:
7+
- master
8+
9+
jobs:
10+
test_freethreading:
11+
runs-on: ubuntu-latest
12+
strategy:
13+
fail-fast: false
14+
matrix:
15+
python-version: ["3.13t", "3.14t"]
16+
17+
steps:
18+
- uses: actions/checkout@v6
19+
with:
20+
submodules: recursive
21+
22+
- name: Install uv
23+
uses: astral-sh/setup-uv@v6
24+
25+
- name: Set up free-threaded Python ${{ matrix.python-version }}
26+
run: uv python install ${{ matrix.python-version }}
27+
28+
- name: Install OpenBLAS
29+
run: |
30+
sudo apt-get update
31+
sudo apt-get install -y libopenblas-dev
32+
33+
- name: Create venv and install dependencies
34+
run: |
35+
uv venv --python ${{ matrix.python-version }} .venv
36+
source .venv/bin/activate
37+
uv pip install numpy scipy pytest pytest-run-parallel
38+
39+
- name: Build and install scs
40+
run: |
41+
source .venv/bin/activate
42+
uv pip install -v .
43+
44+
- name: Run tests
45+
run: |
46+
source .venv/bin/activate
47+
python -m pytest test/ -v
48+
49+
- name: Run tests in parallel threads (race detection)
50+
run: |
51+
source .venv/bin/activate
52+
python -m pytest test/ -v --parallel-threads=4 --iterations=3
53+
54+
test_sanitizers:
55+
runs-on: ubuntu-latest
56+
strategy:
57+
fail-fast: false
58+
matrix:
59+
include:
60+
- sanitizer: tsan
61+
cpython_configure_flag: "--with-thread-sanitizer"
62+
cflags: "-fsanitize=thread -g"
63+
- sanitizer: asan
64+
cpython_configure_flag: "--with-address-sanitizer"
65+
cflags: "-fsanitize=address -g"
66+
67+
steps:
68+
- uses: actions/checkout@v6
69+
with:
70+
submodules: recursive
71+
72+
- name: Install build dependencies
73+
run: |
74+
sudo apt-get update
75+
sudo apt-get install -y clang libopenblas-dev libssl-dev zlib1g-dev \
76+
libbz2-dev libreadline-dev libsqlite3-dev libncurses5-dev \
77+
libncursesw5-dev xz-utils libffi-dev liblzma-dev
78+
79+
- name: Cache CPython ${{ matrix.sanitizer }} build
80+
id: cache-cpython
81+
uses: actions/cache@v4
82+
with:
83+
path: cpython-${{ matrix.sanitizer }}
84+
key: cpython-${{ matrix.sanitizer }}-3.14-${{ runner.os }}-v1
85+
86+
- name: Build CPython 3.14t with ${{ matrix.sanitizer }}
87+
if: steps.cache-cpython.outputs.cache-hit != 'true'
88+
run: |
89+
git clone --depth 1 https://github.com/python/cpython.git -b 3.14 cpython-src
90+
cd cpython-src
91+
CC=clang CXX=clang++ ./configure --disable-gil ${{ matrix.cpython_configure_flag }} \
92+
--prefix $GITHUB_WORKSPACE/cpython-${{ matrix.sanitizer }}
93+
make -j$(nproc)
94+
make install
95+
96+
- name: Create venv and install dependencies
97+
run: |
98+
$GITHUB_WORKSPACE/cpython-${{ matrix.sanitizer }}/bin/python3.14t -m venv .san-venv
99+
source .san-venv/bin/activate
100+
pip install numpy scipy pytest meson-python meson ninja
101+
102+
- name: Set sanitizer runtime options
103+
run: |
104+
if [ "${{ matrix.sanitizer }}" = "tsan" ]; then
105+
echo "TSAN_OPTIONS=halt_on_error=1 allocator_may_return_null=1 suppressions=${{ github.workspace }}/test/tsan-suppressions.txt" >> $GITHUB_ENV
106+
echo "OPENBLAS_NUM_THREADS=1" >> $GITHUB_ENV
107+
elif [ "${{ matrix.sanitizer }}" = "asan" ]; then
108+
echo "ASAN_OPTIONS=halt_on_error=1 allocator_may_return_null=1" >> $GITHUB_ENV
109+
echo "LSAN_OPTIONS=suppressions=${{ github.workspace }}/test/lsan-suppressions.txt" >> $GITHUB_ENV
110+
fi
111+
112+
- name: Build and install scs
113+
run: |
114+
source .san-venv/bin/activate
115+
pip install -v . --no-build-isolation
116+
env:
117+
CC: clang
118+
CFLAGS: ${{ matrix.cflags }}
119+
120+
- name: Run tests under ${{ matrix.sanitizer }}
121+
run: |
122+
source .san-venv/bin/activate
123+
python -m pytest test/ -v -s
124+
125+
- name: Run threading stress tests under ${{ matrix.sanitizer }}
126+
run: |
127+
source .san-venv/bin/activate
128+
python -m pytest test/test_free_threading.py test/test_thread_safety.py -v -s

‎meson.build‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,7 @@ if get_option('native_arch')
132132
endif
133133

134134
_deps = [blas_deps]
135+
_deps += dependency('threads')
135136
if get_option('use_openmp')
136137
_deps += dependency('openmp')
137138
endif

‎pyproject.toml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ classifiers = [
2323
'Programming Language :: Python :: 3.13',
2424
'Programming Language :: Python :: 3.14',
2525
'Programming Language :: Python :: 3 :: Only',
26+
'Programming Language :: Python :: Free Threading :: 3 - Stable',
2627
'Programming Language :: Python :: Implementation :: CPython',
2728
'Operating System :: Microsoft :: Windows',
2829
'Operating System :: POSIX',
@@ -96,3 +97,11 @@ before-all = [
9697
testpaths = [
9798
"test",
9899
]
100+
faulthandler_timeout = 600
101+
faulthandler_exit_on_timeout = true
102+
markers = [
103+
"thread_unsafe: mark test as unsafe to run in parallel threads (pytest-run-parallel)",
104+
]
105+
# pytest-run-parallel: functions that are not thread-safe
106+
# (capsys/capfd are detected automatically by pytest-run-parallel)
107+
thread_unsafe_functions = ["numpy.random.seed"]

‎scs/py/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,11 @@ def __init__(self, data, cone, **settings):
9595
@param data Dictionary containing keys `P`, `A`, `b`, `c`.
9696
@param cone Dictionary containing cone information.
9797
@param settings Settings as kwargs, see docs.
98+
99+
Thread safety: construction is assumed to be thread-local. Calling
100+
`__init__` on a live SCS instance from another thread (i.e. while
101+
`solve` or `update` may be running on it) is undefined behavior.
102+
Use a fresh `SCS(...)` instance instead.
98103
"""
99104
self._settings = settings
100105
if not data or not cone:

‎scs/scsmodule.h‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,10 @@ static PyObject *moduleinit(void) {
6464
return NULL;
6565
}
6666

67+
#ifdef Py_GIL_DISABLED
68+
PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);
69+
#endif
70+
6771
/* Initialize SCS_Type */
6872
SCS_Type.tp_new = PyType_GenericNew;
6973
if (PyType_Ready(&SCS_Type) < 0)

0 commit comments

Comments
 (0)