Skip to content

pixi environment support#112

Open
davidrich27 wants to merge 10 commits into
fix-macos-buildfrom
pixi-environment
Open

pixi environment support#112
davidrich27 wants to merge 10 commits into
fix-macos-buildfrom
pixi-environment

Conversation

@davidrich27

@davidrich27 davidrich27 commented Mar 6, 2026

Copy link
Copy Markdown
Contributor

Note: This PR is built on a fork of the fix-macos-build branch and will be awaiting that PR's merge.

Summary

Adds a pixi environment for building larch, providing reproducible builds without system dependencies. All compilers, libraries, and tools come from conda-forge via pixi.

Pixi build workflow

pixi run init                    # creates larch-build.env, inits git submodules
# (optionally edit larch-build.env)
pixi run configure               # runs cmake with settings from larch-build.env
pixi run build                   # builds larch
pixi run test                    # runs tests (excluding slow)

Build settings (build dir, build type, USE_USHER, USE_NETAM, libtorch path, extra cmake flags, job count) are stored in larch-build.env at the project root. This file is gitignored and user-editable between init and configure. A copy is saved into the build directory at configure time for reference.

What's included

  • pixi.toml — workspace config with all dependencies (GCC 13, cmake, protobuf, boost, openmpi, pytorch-cpu, etc.) and task definitions
  • scripts/init.sh, configure.sh, load-config.sh, fetch-libtorch.sh
  • conda environment files — CI and dev environments for Linux and macOS, with and without netam
  • macOS CI workflows — new GitHub Actions for macOS builds
  • CMakeLists.txt changes:
    • macOS compatibility (CONFIG mode protobuf, AppleClang linker flags, -std=c++2b instead of -std=c++23)
    • matplotplusplus GCC 13 warning suppression (-Wno-stringop-truncation -Wno-array-bounds -Wno-stringop-overread)
  • configure.sh passes -DProtobuf_INCLUDE_DIR to ensure pixi's protobuf headers are used instead of system headers
  • Default build type is RelWithDebInfo (matches Ognian's recommended build, and avoids a protobuf ABI issue with Debug builds — see below)

Known issue: protobuf Debug/Release ABI mismatch

When building in Debug mode with conda/pixi protobuf, linking fails with an undefined reference to InternalMetadata::~InternalMetadata(). This is because conda builds protobuf in Release mode (with NDEBUG), which inlines and doesn't export that symbol. Debug builds expect it in libprotobuf.so.

RelWithDebInfo and Release builds work fine (they define NDEBUG). The default build type is set to RelWithDebInfo to avoid this.

Possible solutions if Debug builds are needed in the future:

  1. NDEBUG on protobuf interface — simple, version-agnostic, but disables assert() globally (larch uses its own Assert() macro so impact is minimal)
  2. Build protobuf from source via FetchContent — cleanest long-term fix but adds build time and risks conflicts with libtorch's vendored protobuf

Test results

Tested on both local machine and remote compute cluster — results match.

v0.1.0 4300d (main) pixi (this PR)
Total tests 54 89 140
Failed 3 5 4
Build type Release Release RelWithDebInfo
Compiler GCC 12.4 GCC 12.4 GCC 13.4 (pixi)

Pre-existing failures (all 3 builds): dagbin simultaneous write, dagbin seeded, ambiguous VCF with MatOptimize.

Regression from v0.1.0 → main: MATConversion: Check SampleIds (still fails on this branch — not introduced by pixi).

Fixed on this branch: RF distance: counter (via fix-macos-build base: RFDistanceSumRFDistance).

New tests: 51 netam/ML tests (Netam Likelihood, KmerEncoder, Model, ML SPR, S5F) — all pass.

Test plan

  • Non-netam build and tests (local)
  • Netam build and tests (local) — builds and passes (140 tests, 4 pre-existing failures)
  • Remote cluster build and tests — matches local results
  • macOS build — not yet tested

🤖 Generated with Claude Code

davidrich27 and others added 5 commits March 3, 2026 22:00
- Default build type changed to RelWithDebInfo (avoids protobuf
  Debug/Release ABI mismatch with conda-built libprotobuf)
- Added all team members to pixi.toml authors
- configure.sh passes Protobuf_INCLUDE_DIR to ensure pixi headers
  are used instead of system headers
- Added deps/libtorch to .gitignore
- Added matplotplusplus warning suppression flags for GCC 13
- Updated pixi.lock

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@davidrich27 davidrich27 marked this pull request as ready for review March 6, 2026 06:55
davidrich27 and others added 5 commits March 6, 2026 00:12
- Replaced inline defaults in init.sh with larch-build.env.template
- Added LARCH_PROTOBUF_PATH option for custom protobuf installs
  (enables Debug builds by avoiding conda protobuf ABI issue)
- Simplified init.sh to copy template, load-config.sh to source and export
- Added pixi build instructions to README.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- init.sh: env var overrides applied on top of template defaults
- configure.sh: use bash array for cmake args (safe with spaces)
- configure.sh: platform-aware protobuf lib detection (.so/.dylib)
- configure.sh: replace GNU find -printf with portable alternative
- configure.sh: properly quote nested dirname calls
- load-config.sh: add set -euo pipefail
- Extract test setup into scripts/run-tests.sh (DRY)
- Template: leave LARCH_BUILD_JOBS empty for auto-detect

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split pixi.toml deps into shared/Linux/macOS sections
- Linux: gcc/gxx, ucx, yasm, libhwloc
- macOS: tbb-devel, libprotobuf (compiler from Xcode)
- configure.sh auto-detects macOS and enables USE_SYSTEM_TBB,
  sets Protobuf_ROOT to CONDA_PREFIX
- Updated pixi.lock with macOS package resolutions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@davidrich27 davidrich27 marked this pull request as draft March 19, 2026 23:41
@davidrich27 davidrich27 changed the base branch from main to fix-macos-build March 19, 2026 23:43
@davidrich27 davidrich27 marked this pull request as ready for review March 19, 2026 23:43
@davidrich27 davidrich27 changed the title WIP: pixi environment support pixi environment support Apr 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant