Skip to content

Commit 6d55a99

Browse files
build(docker): cross-compile the musl build instead of emulating it
The scratch runtime as first written dropped the --platform pin from the musl builder, so arm64 compiled under QEMU. I estimated 20-45 minutes for that by extrapolating from Nexus, which builds the same way. Nexus's tree is far smaller; ours ran 6h42m and was killed still compiling. The estimate was presented as informed and was not — it should have been "I have not measured this". Cross-compiled with cargo-zigbuild the same build takes 10 minutes for both architectures. Both numbers are now in the Dockerfile so nobody repeats the emulation thinking it is equivalent. zig rather than clang because Rust links musl targets with rustup's own self-contained CRT, so pure-Rust code cross-compiles unaided — the gap is aws-lc-sys and zstd-sys, which are C. `xx-info triple` resolves musl targets to -linux-gnu, and clang against Debian's arm64 musl sysroot fails on a missing crtend.o. `zig cc` ships a complete cross toolchain, which is exactly what was missing. Three failures on the way, each fixed rather than worked around: - The toolchain shared a stage with the compile, so when crates.io served 503s mid-fetch the whole thing restarted. It is now its own stage, and cargo-zigbuild arrives as the project's prebuilt binary instead of `cargo install --locked`, which compiled a tool from source on the critical path of every build. That stage now takes 20 seconds. - Both architectures shared the cargo registry cache mount without a sharing mode, so they raced unpacking into the same directory and died with "failed to unpack cfg-if: File exists". Now `sharing=locked`. - The static-link gate was written `... || { echo "not statically linked"; }`, which catches every failure in the chain. When the compile failed the binary never existed, and the gate still blamed linkage — a check reporting the wrong cause with confidence. Now an `if !` around only what it tests. Also pins pnpm. The dashboard stage installed `pnpm@latest` over node:20, and pnpm's latest started requiring Node >= 22.13 (it imports node:sqlite), so the build broke with ERR_UNKNOWN_BUILTIN_MODULE. Nothing here changed; the outside world did. Every workflow already pinned pnpm@10 and dashboard/package.json declares packageManager pnpm@10.15.1 — the Dockerfile was the one place still floating. Published and verified: 3.7.1 and 3.7.1-fastembed, both amd64+arm64. The default image scans 0 packages / 0 vulnerabilities and boots healthy from the registry; the arm64 fastembed binary runs (`vectorizer 3.7.1`), which is the check 3.6.0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 775a5f1 commit 6d55a99

1 file changed

Lines changed: 54 additions & 27 deletions

File tree

Dockerfile

Lines changed: 54 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -172,8 +172,16 @@ RUN cargo chef prepare --recipe-path recipe.json
172172
FROM node:20-bookworm AS dashboard-builder
173173
WORKDIR /dashboard
174174

175-
# Install pnpm
176-
RUN npm install -g pnpm@latest
175+
# Install pnpm.
176+
#
177+
# PINNED, and not to `@latest`. Every workflow already pins `pnpm@10` and
178+
# `dashboard/package.json` declares `packageManager: pnpm@10.15.1`; this line
179+
# was the one place still floating, and it broke the moment pnpm's latest
180+
# started requiring Node >= 22.13 — it imports `node:sqlite`, which does not
181+
# exist on the node:20 base, so `pnpm install` died with
182+
# ERR_UNKNOWN_BUILTIN_MODULE. Nothing in this repo changed; the outside world
183+
# did.
184+
RUN npm install -g pnpm@10.15.1
177185

178186
# Copy dashboard files
179187
COPY dashboard/package.json dashboard/pnpm-lock.yaml dashboard/pnpm-workspace.yaml ./
@@ -393,20 +401,52 @@ FROM busybox:stable-musl AS busybox
393401
# to `-linux-gnu` (`xx-info triple`), and pointing clang at Debian's arm64
394402
# musl sysroot fails on missing `crtend.o`, so cross-compiling this would mean
395403
# hand-assembling a toolchain rather than using one.
396-
FROM rust:1.95-slim-trixie AS builder-musl
404+
# ----------------------------------------------------------------------------
405+
# Cross-compilation toolchain — its own stage on purpose
406+
# ----------------------------------------------------------------------------
407+
# Split out so a network hiccup while fetching it cannot invalidate the
408+
# expensive compile layer below, and vice versa. That is not hypothetical: a
409+
# publish attempt died here when crates.io served 503s from its CDN mid-fetch,
410+
# and because the toolchain shared a stage with the build, the whole thing had
411+
# to start over.
412+
#
413+
# `cargo-zigbuild` arrives as the project's own prebuilt binary rather than
414+
# `cargo install --locked`, which compiled it from source on every build —
415+
# minutes of work, and a crates.io dependency, both on the critical path of
416+
# something that is just a tool.
417+
FROM --platform=${BUILDPLATFORM:-linux/amd64} debian:trixie-slim AS zig-toolchain
418+
ARG ZIG_VERSION=0.13.0
419+
ARG CARGO_ZIGBUILD_VERSION=0.23.3
420+
RUN apt-get update && apt-get install -y --no-install-recommends curl xz-utils ca-certificates && rm -rf /var/lib/apt/lists/* && curl -sSL --retry 5 --retry-all-errors "https://ziglang.org/download/${ZIG_VERSION}/zig-linux-x86_64-${ZIG_VERSION}.tar.xz" | tar -xJ -C /opt && mv "/opt/zig-linux-x86_64-${ZIG_VERSION}" /opt/zig && curl -sSL --retry 5 --retry-all-errors -o /tmp/czb.tar.xz "https://github.com/rust-cross/cargo-zigbuild/releases/download/v${CARGO_ZIGBUILD_VERSION}/cargo-zigbuild-x86_64-unknown-linux-musl.tar.xz" && tar -xJf /tmp/czb.tar.xz -C /usr/local/bin --strip-components=1 "cargo-zigbuild-x86_64-unknown-linux-musl/cargo-zigbuild" && rm /tmp/czb.tar.xz && /opt/zig/zig version && cargo-zigbuild --version
421+
422+
# ----------------------------------------------------------------------------
423+
# STATIC MUSL BUILD
424+
# ----------------------------------------------------------------------------
425+
# Pinned to $BUILDPLATFORM and cross-compiled, NOT built per-target. That
426+
# distinction is the whole cost of this stage:
427+
#
428+
# emulated arm64 (no --platform pin) : killed at 6h42m, still compiling
429+
# cross-compiled with zig : 7m05s
430+
#
431+
# Measured, after the first form was tried and abandoned. Nexus builds its
432+
# arm64 under emulation and lives with it; its tree is far smaller than this
433+
# one, so copying that choice without measuring was the mistake.
434+
#
435+
# Why zig rather than clang: Rust links musl targets with rustup's own
436+
# self-contained CRT, so pure-Rust code cross-compiles unaided — but
437+
# `aws-lc-sys` and `zstd-sys` are C and need a cross C compiler. `xx-info
438+
# triple` resolves musl targets to `-linux-gnu`, and clang against Debian's
439+
# arm64 musl sysroot fails on a missing `crtend.o`. `zig cc` ships a complete
440+
# cross toolchain for every target it supports, which is exactly the gap.
441+
FROM --platform=${BUILDPLATFORM:-linux/amd64} rust:1.95-slim-trixie AS builder-musl
397442
WORKDIR /vectorizer
398443

444+
COPY --from=zig-toolchain /opt/zig /opt/zig
445+
COPY --from=zig-toolchain /usr/local/bin/cargo-zigbuild /usr/local/bin/cargo-zigbuild
446+
ENV PATH="/opt/zig:${PATH}"
447+
399448
ARG TARGETARCH
400-
RUN apt-get update \
401-
&& apt-get install -y --no-install-recommends \
402-
musl-tools clang lld cmake protobuf-compiler pkg-config file perl make \
403-
&& rm -rf /var/lib/apt/lists/* \
404-
&& case "${TARGETARCH:-amd64}" in \
405-
amd64) TARGET_TRIPLE=x86_64-unknown-linux-musl ;; \
406-
arm64) TARGET_TRIPLE=aarch64-unknown-linux-musl ;; \
407-
*) echo "unsupported TARGETARCH '${TARGETARCH}'" >&2; exit 1 ;; \
408-
esac \
409-
&& rustup target add "${TARGET_TRIPLE}"
449+
RUN apt-get update && apt-get install -y --no-install-recommends cmake protobuf-compiler pkg-config file perl make && rm -rf /var/lib/apt/lists/* && case "${TARGETARCH:-amd64}" in amd64) TARGET_TRIPLE=x86_64-unknown-linux-musl ;; arm64) TARGET_TRIPLE=aarch64-unknown-linux-musl ;; *) echo "unsupported TARGETARCH '${TARGETARCH}'" >&2; exit 1 ;; esac && rustup target add "${TARGET_TRIPLE}"
410450

411451
ARG PROFILE=release-docker
412452
ARG GIT_COMMIT_ID
@@ -424,20 +464,7 @@ COPY --from=dashboard-builder /dashboard/dist /vectorizer/dashboard/dist
424464
# would build fine here and then fail to exec in `scratch`, where there is no
425465
# loader. `ldd` is NOT usable for this — glibc's ldd prints "statically
426466
# linked" and exits 0 for static-PIE binaries, so it would pass either way.
427-
ENV CARGO_BUILD_JOBS=2
428-
RUN --mount=type=cache,target=/usr/local/cargo/registry \
429-
unset OPENSSL_DIR OPENSSL_INCLUDE_DIR OPENSSL_LIB_DIR OPENSSL_STATIC; \
430-
case "${TARGETARCH:-amd64}" in \
431-
amd64) TARGET_TRIPLE=x86_64-unknown-linux-musl ;; \
432-
arm64) TARGET_TRIPLE=aarch64-unknown-linux-musl ;; \
433-
esac \
434-
&& cargo build --profile "$PROFILE" --package vectorizer-server --bin vectorizer \
435-
--no-default-features --target "$TARGET_TRIPLE" \
436-
&& PROFILE_DIR=$(if [ "$PROFILE" = dev ]; then echo debug; else echo "$PROFILE"; fi) \
437-
&& cp "target/${TARGET_TRIPLE}/${PROFILE_DIR}/vectorizer" /vectorizer-static \
438-
&& file /vectorizer-static | grep -Eq 'static-pie linked|statically linked' \
439-
|| { echo "::error::binary is not statically linked — it would not exec in scratch"; \
440-
file /vectorizer-static; exit 1; }
467+
RUN --mount=type=cache,target=/usr/local/cargo/registry,sharing=locked unset OPENSSL_DIR OPENSSL_INCLUDE_DIR OPENSSL_LIB_DIR OPENSSL_STATIC; case "${TARGETARCH:-amd64}" in amd64) TARGET_TRIPLE=x86_64-unknown-linux-musl ;; arm64) TARGET_TRIPLE=aarch64-unknown-linux-musl ;; esac && cargo zigbuild --profile "$PROFILE" --package vectorizer-server --bin vectorizer --no-default-features --target "$TARGET_TRIPLE" && PROFILE_DIR=$(if [ "$PROFILE" = dev ]; then echo debug; else echo "$PROFILE"; fi) && cp "target/${TARGET_TRIPLE}/${PROFILE_DIR}/vectorizer" /vectorizer-static && if ! file /vectorizer-static | grep -Eq 'static-pie linked|statically linked'; then echo "::error::binary is not statically linked — it would not exec in scratch"; file /vectorizer-static; exit 1; fi
441468

442469
# ============================================================================
443470
# USER PREP — throwaway stage, only text files survive into the runtime

0 commit comments

Comments
 (0)