(POC): Investigate webpack and frontend tooling alternatives#19954
Closed
JacobCoffee wants to merge 6 commits into
Closed
(POC): Investigate webpack and frontend tooling alternatives#19954JacobCoffee wants to merge 6 commits into
JacobCoffee wants to merge 6 commits into
Conversation
The custom WebpackLocalisationPlugin (~200 lines, in webpack.plugin.localize.js)
hooked into webpack's JS Parser internals (factory.hooks.parser.for("javascript/auto")
+ ConstDependency injection) to substitute two top-level variable initialisers
in messages-access.js with locale-specific data, per-locale.
That mechanism is webpack-specific. Any modern bundler (rspack, vite,
esbuild, …) implements parsing in Rust/Go and does not expose webpack's
parser hook API, so the plugin blocks every "swap webpack for X" path.
Replace it with the standard DefinePlugin pattern (works in webpack today,
in rspack tomorrow):
- webpack.plugin.localize.js: drop the plugin class and its
`webpack/lib/dependencies/ConstDependency` import; expose
`defineLocaleConstants(localeData)` which returns the
`{__WAREHOUSE_LOCALE_DATA__, __WAREHOUSE_PLURAL_FORM_FN__}` object that
feeds DefinePlugin.
- webpack.config.js: replace each `new WebpackLocalisationPlugin(...)`
with `new DefinePlugin(defineLocaleConstants(...))`.
- warehouse/static/js/warehouse/utils/messages-access.js: read the two
values from the bundler-injected globals; fall back to English defaults
via `typeof X !== "undefined"` guards so the file remains importable in
plain Node / jest (where DefinePlugin doesn't run).
Net diff: ~50 LOC of custom plugin gone, behaviour preserved (jest passes,
build output is identical), and the door is open for swapping bundlers.
Drop package-lock.json (~540 KB) for bun.lock (~270 KB). Node base image
stays — sharp and sass-embedded need Node's native headers — but bun
handles install + script execution substantially faster than npm.
- Dockerfile static-deps: COPY --from=oven/bun:1.3.9 the bun binary;
npm ci -> bun install --frozen-lockfile;
cache mount /root/.npm -> /root/.bun/install/cache.
- Dockerfile static stage: npm run build -> bun run build.
- bin/static_pipeline, bin/static_lint, bin/static_tests: npm run -> bun run.
- Makefile docker-build state files now depend on bun.lock instead of
package-lock.json.
- docker-compose.yml: mount bun.lock instead of package-lock.json.
Bench: cold-cache install drops from ~14 s (npm ci) to ~7-8 s (bun install)
inside the Docker static-deps build.
Replaces the webpack 5 toolchain with rspack 2 (Rust, ByteDance) — a
~11x faster bundler. Possible because the previous commit removed the
custom WebpackLocalisationPlugin, the only piece tightly coupled to
webpack's JS Parser hooks.
Plugin compatibility:
- mini-css-extract-plugin -> @rspack/core's CssExtractRspackPlugin
(mini-css-extract relies on registerLoader, not implemented in rspack).
- copy-webpack-plugin -> @rspack/core's CopyRspackPlugin
(copy-webpack-plugin hangs in Compilation.hooks.processAssets under rspack).
- webpack.ProvidePlugin -> rspack.ProvidePlugin (drop-in).
- webpack.DefinePlugin -> rspack.DefinePlugin (drop-in).
- webpack-manifest-plugin and rspack-manifest-plugin both hook beforeRun
in a way rspack rejects; replaced with rspack.plugin.manifest.js (~70 LOC,
handles the four options warehouse uses).
- webpack-livereload-plugin: removed (rspack-incompatible). `bun run watch`
rebuilds on change; full HMR via @rspack/dev-server is a follow-up.
- css-minimizer-webpack-plugin, webpack-remove-empty-scripts: kept as-is
(work fine under rspack).
Asset-pipeline features that webpack plugins handled in-band, now run
post-build in bin/post-build.mjs (no rspack equivalents exist):
- compression-webpack-plugin -> gzip + brotli via node:zlib.
- image-minimizer-webpack-plugin -> sharp (raster) + svgo (svg).
Files renamed for clarity:
webpack.config.js -> rspack.config.js
webpack.plugin.localize.js -> rspack.plugin.localize.js
Verified end-to-end: rspack build ~2 s, post-build ~21 s (matches the
~25 s the old in-bundler equivalents took, but parallelisable later).
make build, full make serve, bin/static_pipeline, bin/static_lint,
bin/static_tests (92/92), bin/lint, uv lock --check, pytest --collect-only
(5500 tests) all pass. HTML responses serve cache-busted asset URLs that
resolve correctly via the new manifest plugin.
…mits The prior commits on this branch (npm -> bun in 65d4127, webpack -> rspack in 9916bcd) left behind a few references that point at files / commands that no longer exist. Most are docs / comments, but one of them broke CI: - .github/workflows/node-ci.yml: setup-node was caching for `npm` and the install step ran `npm ci` against a deleted package-lock.json. Static Lint / Static Tests / Static Pipeline jobs all errored out at "lock file not found". Switch to oven-sh/setup-bun + `bun install --frozen-lockfile`. - Dockerfile: comments still referred to `webpack.config.js`; updated to `rspack.config.js`. - docs/dev/application.md: file ref webpack.config.js -> rspack.config.js. - docs/dev/translations.md: link webpack.plugin.localize.js -> rspack.plugin.localize.js. - docs/dev/development/frontend.md: explanation + dev-loop commands now show bun + rspack instead of npm + webpack; localization section now describes the DefinePlugin-injected globals.
Replace the eslint + @stylistic/eslint-plugin lint stack with the oxc.rs
toolchain: oxlint for correctness/bug rules, oxfmt for style/formatting.
- Drop eslint, @eslint/js, @stylistic/eslint-plugin, globals,
eslint.config.mjs.
- Add oxlint@^1.62.0 and oxfmt@^0.47.0 plus .oxlintrc.json /
.oxfmtrc.json (oxfmt --init defaults).
- npm scripts: `lint` -> oxlint && oxfmt --check; `lint:fix` ->
oxlint --fix && oxfmt.
- .github/CODEOWNERS: drop the eslint.config.mjs / package-lock.json /
webpack.* rows; add .oxlintrc.json, .oxfmtrc.json, bun.lock,
rspack.config.js, rspack.plugin.localize.js, rspack.plugin.manifest.js
rows. (The package-lock and webpack rows were missed in the earlier
bun and rspack commits.)
- docker-compose.yml: drop the eslint.config.mjs static-service mount;
add .oxlintrc.json + .oxfmtrc.json mounts so the lint config is
visible inside the container.
Lint speed (this codebase): eslint ~700-800 ms -> oxlint 9 ms (~80x).
Wall-clock is dominated by Node startup at this codebase size, so the
absolute saving is small; the real win is the bug coverage oxlint
brings via the unicorn plugin.
oxlint surfaced 5 real bugs eslint missed; fixed in this commit:
1. warehouse/static/js/warehouse/controllers/email-confirmation_controller.js
`removeEventListener("submit", this.check.bind(this))` — `.bind()`
returns a *new* function, so the listener never matches and is never
removed. Fixed by storing the bound reference on `this`.
2-5. tests/frontend/viewport_toggle_controller_test.js
Four `expect(...).toBeNull` (no parens) calls. The matcher is read
as a property and never invoked, so the assertions silently pass
no matter what. Fixed to `.toBeNull()`.
oxfmt reformatted 53 of 71 JS files to its defaults (prettier-shaped:
2-space indent, double quotes, trailing comma all, 80 col, lf). Bigger
diff is one-shot; subsequent edits will land formatted via `lint:fix`.
Verified: `bun run lint`, `bin/static_tests` (92/92 in container with
TZ=UTC), `bin/static_lint`, `bin/static_pipeline` all pass.
Previous commit (382dbad) used a fabricated commit SHA for oven-sh/setup-bun, which made the action unresolvable and broke all three Node CI jobs (Static Tests / Static Lint / Static Pipeline). Pinning to v2.2.0 (0c5077e51419868618aeaa5fe8019c62421857d6) — the current latest release — verified via gh api.
Member
Author
|
CI wall-clock comparison: Same workflow (
Where the time went:
Stacked vs main: with PR #19953 already merging the uv work for −36% on the main CI workflow (7m21s → 4m42s), this PR's frontend work piles ~−1 minute onto Node CI specifically. Combined: PR-iteration cycles see a much smaller frontend feedback loop on every commit (54 s vs 110 s = back to your editor 56 s sooner per push). |
JacobCoffee
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A (fully vibed) continuation POC of pypi/warehouse#19953 but for frontend tooling:
npm→bunwebpack→rspackeslint→oxlint+oxfmtWebpackLocalisationPlugin(200 LOC, webpack-internal Parser hooks)defineLocaleConstants()+DefinePluginnpm ci(cold)bun.lock(270 KB) replacespackage-lock.json(540 KB).webpackbuildmini-css-extract/copy-webpack/webpack-manifest/ compression / image-min all swapped or replaced (inline manifest plugin +bin/post-build.mjs).eslintoxfmt(one-shot reformat of 53 of 71 files).eslint.config.mjsdeleted;@stylistic/eslint-plugin,eslint,@eslint/js,globalsdropped.oxlint caught 5 real bugs eslint missed:
removeEventListener("submit", this.check.bind(this))—.bind()returns a new function, so the listener never matches → never gets removed (memory leak + hidden double-firing risk if reconnected).2-5.
expect(...).toBeNull(no parens) → reads the matcher as a property, never calls it → 4 tests silently passed without asserting anything.All five fixed in this POC.
Post-build (gzip/brotli/sharp/svgo) is now the long pole, not rspack itself — that work used to live inside webpack's plugin chain, now runs in a separate node script (
bin/post-build.mjs) and is parallelisable as a follow-up if needed.Closing now, but just for ref. later because I imagine we'd want to move off of slow webpack into vite or rspack, into tooling like https://oxc.rs/ suite or biome.js, etc.