Skip to content

ci(llvm): reduce Windows LLVM tool count while keeping llvm-config (retry of reverted tool-disable) #364

@nebasuke

Description

@nebasuke

Context

On Windows hosted runners, lld-link is slow per-executable. With ccache at 100% hit on compiles, the dominant wall-clock in build-llvm is linking the ~200 LLVM tool binaries (opt.exe, llc.exe, llvm-objdump.exe, etc.). solx consumes LLVM purely as a library via inkwell FFI, so none of these tool binaries are invoked at runtime.

An attempt to disable them via LLVM_BUILD_TOOLS=Off + LLVM_INCLUDE_TOOLS=Off (commit 3106d1e in #360) was reverted because it broke the Rust test step.

Evidence the approach works (and what broke)

Test run #24771100328 at commit 3106d1e:

  • Build LLVM step on Windows: 42 min → 9:44 (4.3×). The tool-disable mechanism worked exactly as hypothesised — ~200 fewer lld-link invocations.
  • Run tests step failed with:
    error: No suitable version of LLVM was found system-wide or pointed
    error: could not compile `llvm-sys` (lib) due to 1 previous error
    

Root cause: the llvm-sys Rust crate (pulled in by inkwell) runs ${LLVM_SYS_211_PREFIX}/bin/llvm-config at Rust crate-build time to discover include paths, library paths, and link flags. llvm-config is an LLVM "tool" under llvm/tools/llvm-config/, so LLVM_INCLUDE_TOOLS=Off drops its CMakeLists from the configure and the binary isn't produced.

llvm-config is the only tool binary solx needs (and only at Rust build time, not runtime). Every other tool we disabled is safely unused.

Proposed approaches

Option A — surgical override

Keep LLVM_INCLUDE_TOOLS=On (so tools/ subdirectory is configured) but LLVM_BUILD_TOOLS=Off (so tools aren't in the ALL target by default), then force just llvm-config:

-DLLVM_BUILD_TOOLS='Off'
-DLLVM_INCLUDE_TOOLS='On'
-DLLVM_TOOL_LLVM_CONFIG_BUILD='On'

LLVM's cmake uses the LLVM_TOOL_<name>_BUILD pattern to allow per-tool overrides of LLVM_BUILD_TOOLS. Expectation: llvm-config.exe is built, the ~200 other tool binaries are not.

Risks to validate in CI:

  1. LLVM_INCLUDE_TOOLS=On re-emits configure/compile rules for all tools. If LLVM's cmake uses EXCLUDE_FROM_ALL correctly, ninja only builds llvm-config in practice. If not, the speedup partially erodes. Needs measurement.
  2. Some tools may be tangled in install targets (e.g. via llvm_install_symlink), which could force them in via install-time dependencies. Watch the final ninja install step for hidden work.

Option B — LLVM_DISTRIBUTION_COMPONENTS whitelist

Most surgical. Tell cmake: "when building the distribution target, include only these components." Shape roughly:

-DLLVM_INCLUDE_TOOLS='On'
-DLLVM_BUILD_TOOLS='Off'
-DLLVM_DISTRIBUTION_COMPONENTS='llvm-libraries;llvm-headers;cmake-exports;llvm-config;mlir-libraries;mlir-headers;lld-libraries'

Requires switching solx-dev llvm build's final step from ninja install to ninja install-distribution. That's a ~one-line change in solx-dev/src/llvm/build.rs or equivalent. Clean but touches Rust code.

Option C — scope to Windows only (same as the reverted commit)

Whichever of A or B lands should retain the Windows-only gate — other platforms weren't link-bound and had no issues with tools enabled.

Expected impact

Based on the successful half of run #24771100328:

Metric Before With tool-disable Speedup
Windows Build LLVM step ~42 min ~9:44 4.3×
Windows whole job ~1h 19min ~15–20 min est. ~4×

Applied per Windows CI run on submodule bumps + any path where ccache is the only cache in play.

Acceptance criteria

  • Windows Build LLVM step completes in under 15 min on a warm-ccache run.
  • Run tests step on Windows completes successfully (verifies llvm-sys found llvm-config).
  • No regression on Linux/macOS legs (confirmed by wall-clock comparison against the last good run).
  • Show ccache stats still reports near-100% hit rate on the warm path.

Out of scope

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions