feat(catalog): filesystem-backed LOCAL catalog mode#4938
Open
Austin-s-h wants to merge 36 commits into
Open
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## feat/preview-frontend #4938 +/- ##
==========================================================
- Coverage 47.74% 33.85% -13.90%
==========================================================
Files 819 733 -86
Lines 34123 27462 -6661
Branches 5780 5806 +26
==========================================================
- Hits 16293 9296 -6997
- Misses 15859 16191 +332
- Partials 1971 1975 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+84
to
+91
| def _build_env(self) -> dict[str, str]: | ||
| env = dict(os.environ) | ||
| env.update(settings.local_backend_env()) | ||
| env["QUILT_REPO_ROOT"] = str(self.repo_root) | ||
| # Make quilt3_local.voila_kernel importable inside the voila env. | ||
| api_python = str(self.repo_root / "api" / "python") | ||
| existing = env.get("PYTHONPATH", "") | ||
| env["PYTHONPATH"] = f"{api_python}{os.pathsep}{existing}" if existing else api_python |
Contributor
There was a problem hiding this comment.
_pick_free_port
The socket is closed before Voila actually binds to the port, leaving a window where another process (or even a parallel Voila restart) could claim the same port. In practice this is rare on a developer machine, but it could produce a confusing startup failure where Voila exits immediately and the proxy returns 404 indefinitely. The standard mitigation is to pass SO_REUSEPORT, bind, and keep the socket alive until Voila has started — or to accept the small race and simply retry on bind failure in the polling loop.
Collaborator
Author
There was a problem hiding this comment.
Fixed in eff042c using the retry approach you suggested — start() now relaunches on a fresh port (up to _PORT_RETRIES) when Voila exits immediately, which is the symptom of losing the bind race. That avoids holding the probe socket open across the handoff while still recovering from a collision cleanly.
Austin-s-h
force-pushed
the
feat/local-catalog
branch
2 times, most recently
from
eff042c to
fca79ab
Compare
Re-internalize a LOCAL catalog development mode under api/python/quilt3_local, stacked on the packaging-modernization and preview-feature PRs. Backend (api/python/quilt3_local): * FastAPI app serving config.json, GraphQL (ariadne), search, an s3proxy, and a /__lambda mount. * Two object backends — `aws` (real S3 via temporary credentials) and `filesystem` (a local directory tree), selected via QUILT_LOCAL_OBJECT_BACKEND. * Isolated uv-subprocess lambda runner (lambda_runner.py / lambda_subprocess.py) that runs the preview/tabular/thumbnail/ transcode lambdas in their own environments. * Interactive Voila dashboards behind the opt-in quilt3[local-voila] extra; dormant (graceful no-op) when the flag is off or the extra is not installed. * quilt3.main: expand `catalog` help to describe the two object backends. * Removes the stale top-level quilt3_local/README.md pointer. Frontend (LOCAL-mode plumbing): * Signer: force the s3proxy in LOCAL mode. * Buckets / Overview / Subscription: LOCAL bucket normalization, LOCAL Overview query, and subscription pause in LOCAL mode. CI: add a `test-local-catalog` job that installs quilt3[catalog] and runs tests/test_local_mode.py. Verified locally: test_local_mode.py 30 passed / 1 skipped (incl. a Voila render test); frontend specs 15 passed; live smoke — booted the filesystem backend and confirmed /config.json returns mode=LOCAL and /__reg/api/search is mounted and validating. api/python `poe lint` clean; catalog eslint/prettier clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* webpack.dev: suppress the noisy "ResizeObserver loop" runtime-error overlay in dev. * webpack.base: allow ts-loader to process katex, disable ForkTsChecker in dev-server mode. * husky pre-commit: resolve the catalog dir robustly and skip gracefully when lint-staged isn't installed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…URLs - voila_subprocess: retry Voila startup on a fresh port when it exits immediately, mitigating the _pick_free_port TOCTOU race (per review) without trying to hold the probe socket across the handoff. - build_render_url now strips AWS credential params (access_key/secret_key/ session_token) from the render URL so they can't surface in access logs, browser history, or proxy logs; they are injected into the per-session kernel env by QuiltKernelManager instead. Documented that moving the frontend's URL assembly to a token-keyed store is the intended follow-up. Added test_voila_build_render_url_strips_credentials. - docs/CONTRIBUTING.md: wrap the long local-catalog test command to satisfy markdownlint MD013 (lint-docs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Austin-s-h
force-pushed
the
feat/local-catalog
branch
from
fca79ab to
d5e3bd6
Compare
2 tasks
Austin-s-h
force-pushed
the
feat/preview-frontend
branch
from
b4f7405 to
6231088
Compare
Austin-s-h
force-pushed
the
feat/local-catalog
branch
from
a2a7152 to
e9c5d35
Compare
The `quilt3 catalog` docstring was updated to document the aws/filesystem object backends, but docs/api-reference/cli.md was not regenerated, so the gendocs-check CI job (gendocs + git diff --exit-code) failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
10 tasks
The test-local-catalog job referenced `--group local-catalog-test`, which was never defined in api/python/pyproject.toml, so uv aborted with "Group `local-catalog-test` is not defined". Define it with the test runner (pytest/pytest-env/pytest-cov); runtime deps (graphql/aiohttp/starlette/ asgiproxy) come from the `catalog` extra the job already passes. Kept out of `dev` because `dev`'s quilt3[pyarrow,anndata] conflicts with `--extra catalog`. Also exclude tests/test_local_mode.py from the default poe test/test-cov/ test-verbose commands: test-client runs the default suite without the catalog extra, so its unconditional `from graphql import graphql` broke collection of the entire client suite (548 tests) on every OS/Python combo. The dedicated test-local-catalog job runs that file directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adding the `local-catalog-test` dependency group to api/python/pyproject.toml left three lockfiles that embed quilt3's metadata stale: the root workspace lock (also covers the iceberg/s3hash members) plus gendocs and testdocs. Their `uv run --locked` / export --locked steps fail-fast on a stale lock. Metadata-only change — no dependency versions moved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Condense over-explanatory multiline comments (Voila proxy mount ordering, voila_stub 404 contract, _same_python, subprocess launch rationale, FCS preallocated-array note) while preserving the load-bearing "why". Comment-only; no code or behavior changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two fixes for running the LOCAL catalog UI from a repo checkout:
- main.py: add a /config.js route emitting
`window.QUILT_CATALOG_CONFIG = {…}`. The catalog index.html loads this
synchronously to bootstrap config (production generates it from config.json
in the Dockerfile); built bundles don't ship it, so without the route the
SPA fallback returned index.html and the app failed to init with
"Unexpected token '<'". The original POC generated catalog/static-dev/
config.js; the reorg dropped that.
- run_local_catalog.py: kill any catalog already bound to the port before
launching (stale backgrounded uvicorns were piling up on :3000 and silently
shadowing new launches), auto-detect a built bundle (catalog/build, else the
installed package's catalog_bundle) so the UI isn't blank, trap SIGTERM for
graceful shutdown, and line-buffer stdout. Add --keep-existing to opt out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SmartS3 proxy rewrite did `new AWS.Endpoint(cfg.s3Proxy)`, which assumes an absolute URL with a host. In LOCAL mode cfg.s3Proxy is the relative path "/__s3proxy", so AWS.Endpoint produced an empty host and the presigned URL came out as `https:///__s3proxy/...` — which the browser resolved to the raw `https://<bucket>.s3.<region>.amazonaws.com/...`. The preview/thumbnail/tabular lambdas then fetched a nonexistent bucket and got HTTP 301 → every preview 500'd. Resolve a relative cfg.s3Proxy against window.location.origin before handing it to AWS.Endpoint; absolute proxy URLs (production) are unchanged. Verified in a browser against a filesystem LOCAL catalog: preview/fcs/tabular/pdf/transcode previews now render (lambda calls hit /__s3proxy and return 200). Remaining thumbnail 500s are environmental (libreoffice for pptx, bioio plugin for jpeg). Also condense the `poe catalog-health` probe into a compact table-driven loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pptx→pdf step hardcoded the "libreoffice" command, which is the binary
name on Lambda/Linux but not under Homebrew on macOS (where it's only
installed as "soffice") — so .pptx thumbnails failed locally with
"Missing required command: libreoffice". Resolve via
shutil.which("libreoffice") or shutil.which("soffice"); same binary either way,
prod behavior unchanged. Verified: in.pptx thumbnail returns a 256x362 JPEG
against the LOCAL catalog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the single FCS scatter with a canonical flow-cytometry gating grid: Cells (FSC-A×SSC-A), Singlets (FSC-H×FSC-A), then available fluorescence pairs, capped at 6 panels and emitted only for channel pairs present in the file. Axis titles prefer the $PnS marker (e.g. "CD3 (FL1-A)"), falling back to the detector name and skipping redundant marker==channel labels. Single-channel files keep the original brush-select single panel. Add FCS sample files from tlnagy/fcsexamples, downsampled to ~5k events for the repo (accuri-c6, attune, bd-facs-aria-ii, beckman-cyan, cytof-day3, millipore-easycyte) plus the unmodified 100k-event bd-facs-aria-ii-100k for a realistic-scale preview; staged into the LOCAL demo catalog. Tests cover the multi-panel grid and marker-label paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…px grid
The Perspective datagrid had a fixed minHeight (480/960px), so a 2-row jsonl/csv
rendered a large empty grid below the data ("many lines after the file ends").
For fully-loaded previews with <=20 rows, set an explicit content-based height
(~30px/row, capped at 480px) instead of the tall floor — an explicit height is
required because perspective-viewer is a flex child that collapses to 0 without
one. Larger or truncated datasets keep the tall scrollable viewport.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tion FCS (lambdas/shared/tests/test_preview.py): add direct unit tests for the new gating logic — _select_fcs_panels (canonical order, FCS_MAX_PANELS cap, fallback to first two columns), _fcs_channel_markers ($PnN/$PnS parsing, blank/sentinel filtering, upper/lower-case keys), _axis_label (marker label, marker==channel dedup, None markers), single-panel brush params, and the multi-panel concat/columns/resolve structure with marker propagation. Catalog: extract the relative-vs-absolute proxy resolution from SmartS3 into an exported `resolveProxyUrl(s3Proxy, origin)` helper and cover it with S3.spec.ts (relative -> origin-prefixed, absolute -> untouched, default origin). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Assert /config.js is served as application/javascript (not the SPA HTML fallback), begins with `window.QUILT_CATALOG_CONFIG = `, and that its payload equals /config.json — guarding the regression where the missing route blanked the catalog with "Unexpected token '<'". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the missing top-level CHANGELOG lines for this PR: filesystem-backed LOCAL catalog mode, the presigned-URL s3proxy fix, small-tabular-preview sizing, the multi-panel FCS gating grid, and the headless LibreOffice pptx fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two CI failures surfaced by the FCS gating + LOCAL-catalog work:
- test-lambda(preview): preview's own test_fcs asserted a top-level
vegaLite.title, but multi-panel gating specs title each sub-panel under
`concat`. Accept either shape.
- test-local-catalog: the two Voila-ENABLED tests forced voila_available()
true and mounted the proxy, which imports asgiproxy unconditionally —
absent in CI's `--extra catalog` env (asgiproxy is a local-voila dep and was
only present locally by accident). Make voila_available() also require
asgiproxy so the gate is honest (never mount a proxy whose import would
crash), guard the two proxy-mounting tests with importorskip("asgiproxy"),
and update _simulate_voila_installed + the gate unit test to cover asgiproxy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test-client's test-cov ignores test_local_mode.py (it needs the catalog extra), so quilt3_local/* read as 0% covered and failed the codecov checks. Run the local-catalog job with coverage and upload it under the api-python flag so codecov merges it into that report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Vega-Lite ignores `width: container` inside a concat composition, so each gridded FCS panel collapsed to the ~200px default and rendered skinny and cramped. Give grid cells explicit 300x300 dimensions plus spacing; keep the responsive container width for the single-panel (brush-select) case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The local catalog spawns these lambdas via `uv run --project`, which only resyncs on a version change. With a non-editable path dep, edits to ../shared never propagated to the running lambda without a version bump. Mark the source editable so local edits are live. Dev-only: production zips export with --no-emit-local and pip-install the package separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each module in the LOCAL backend package now has a one-line docstring describing its role (object access, graphql, search, s3proxy, etc.). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Quick start section covering `uv run poe catalog` (one-command browseable LOCAL catalog), which the docs did not mention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Re-internalizes a LOCAL catalog development mode under
api/python/quilt3_local, soquilt3 catalogcan serve a working catalog from a local directory tree without an AWS stack — plus a one-command launcher, health probe, and a round of fixes that make the full preview surface actually render in LOCAL mode.Backend (
api/python/quilt3_local)config.json,config.js(thewindow.QUILT_CATALOG_CONFIGbootstrap the SPA loads), GraphQL (ariadne), search, an s3proxy, and a/__lambdamount.aws(real S3 via temporary credentials) andfilesystem(a local directory tree), selected viaQUILT_LOCAL_OBJECT_BACKEND.lambda_runner.py/lambda_subprocess.py) that runs the preview / tabular / thumbnail / transcode lambdas in their own environments.quilt3[local-voila]extra — dormant (graceful no-op) when the flag is off or the extra isn't installed.quilt3.main:_launch_local_catalognow defaultsQUILT_LOCAL_ORIGINto the boundhost:port, soquilt3 catalog --port <non-3000>no longer breaks the lambda proxy-URL validation (every preview would otherwise 500).Developer tooling
scripts/run_local_catalog.py+poe catalog— one command that stages curated demo fixtures into a local bucket, runs in filesystem mode (no AWS creds), auto-detects a built catalog bundle, kills any catalog already on the port, and shuts down cleanly on SIGTERM.poe catalog-health— a curl-based[OK]/[FAIL]probe across server / config.js / SPA bundle / GraphQL / s3proxy / each preview lambda.Frontend (LOCAL-mode fixes)
utils/AWS/S3.jsx): resolve a relativecfg.s3Proxy(/__s3proxy) againstwindow.location.originbeforeAWS.Endpoint, so presigned preview URLs proxy correctly instead of hitting raws3.amazonaws.com(→ 301 → blank previews). Extracted asresolveProxyUrl.lambdas/shared+renderers/Fcs,renderers/Vega): replace the single scatter with a canonical multi-panel gating grid (Cells → Singlets → fluorescence pairs, capped at 6), with$PnSmarker labels (CD3 (FL1-A)); fix the Vega container width sowidth: "container"specs actually render.renderers/Perspective): small fully-loaded previews size to content instead of a fixed 480px grid (no more huge empty area under a 2-row jsonl).Signer/Buckets/Overview/Subscription: LOCAL bucket normalization, LOCAL Overview query, subscription pause in LOCAL mode.Lambdas
libreofficeorsoffice(Homebrew/macOS) and run it--headlessso.pptxthumbnails work locally without popping a GUI app.Demo data
bd-facs-aria-ii-100k, staged into the LOCAL demo catalog underpreview/scientific/fcs/.CI
Adds a
test-local-catalogjob that installsquilt3[catalog]and runstests/test_local_mode.py(gated on the catalog extra so the default matrix stays green); thelocal-catalog-testdependency group + lock propagation land in this stack.Validation
test_local_mode.py: 34 passed, 1 skipped with--extra catalog(incl./config.jsbootstrap, port→origin propagation, Voila render).lambdas/sharedpreview suite: FCS gating selection / marker / multi-panel structure tests pass.S3.spec.ts(proxy URL resolution) + existing Preview specs pass./__s3proxy.api/pythonpoe lintclean; catalog eslint/prettier clean.TODO
test_local_mode.py+lambdas/sharedFCS tests +S3.spec.ts) + dedicated CI jobdocs/Catalog/LocalMode.md,docs/CONTRIBUTING.md)🤖 Generated with Claude Code