feat(data-marts): Data Last Updated Date

[laskevych]

Preview

Data Mart → Overview

Model → Canvas

MCP (Claude)

Data Mart → Run History - MCP

Summary

Adds Data Last Updated Date: when the source tables behind a Data Mart last changed in the warehouse, surfaced both in the UI and through MCP.

• UI — a tile with a Check now button on the Data Mart page, a sortable column in the Data Marts list, and a badge per node plus a sweep button on the model canvas.
• MCP — a data_last_updated block on query_data_mart, measured live alongside the rows and totals.

Checking reads warehouse metadata only: no consumption, no run recorded.

Scope is the increment agreed in the meeting: MCP first, BigQuery first, best effort, no cache, no separate consumption — plus the UI phase. The Extension and non-BigQuery storages are deliberately out of scope; nothing here is a one-way door.

How it works

Measured on the fly, tied to a run. For MCP this is a third parallel track alongside the rows read and the totals, mirroring ReportTotalsService. The warehouse is already spun up to return data, so the metadata rides along for free and is the freshest possible value at that moment.

A per-storage resolver family (SOURCE_DATA_LAST_UPDATED_RESOLVER + TypeResolver), orchestrated by SourceDataLastUpdatedService. A storage with no implementation resolves to nothing and the caller reports unavailable, so storages can land one at a time.

BigQuery resolution. A free dry run of the composed SQL yields statistics.query.referencedTables, which BigQuery has already expanded through views — nested ones included, up to its own 16-level limit. That satisfies the "View / SQL, 5 layers" requirement natively, with no SQL parsing. Each referenced table's lastModifiedTime then comes from a free metadata call. Sharded sets collapse into a single __TABLES__ rollup per (dataset, prefix) rather than one call per shard.

Persistence. The last-known snapshot lives on data_mart.dataLastUpdated, so lists and the canvas render without touching the warehouse on every page load. The refresh endpoint builds SQL from the Data Mart's own definition, so a persisted value always means "the sources of THIS Data Mart"; the wider blended measurements from MCP journal into their run record and never overwrite it.

Design decisions & tradeoffs

1. A separate resolver family, not job statistics from the main query and not a reader method. Three reasons: (a) latency — the dry run plus metadata calls start at the same moment as the main query and finish first, whereas the main job's statistics only exist after the first batch and would add serial delay to the tail; (b) blending coverage — measuring against the composed SQL means a blended result reports every joined Data Mart's tables, which is where the meeting located the value; (c) reuse — the canvas sweep needs this metadata without reading data and without consumption, which is only possible if the service is decoupled from the reader.

2. The resolver contract is batch-first (resolveForSqlBatch; a single lookup is a batch of one). The expensive part of a lookup is per-storage — resolving credentials and standing up a warehouse client — and a canvas is already filtered to one storage, so a sweep pays that once instead of once per Data Mart. Tradeoff: an MCP query still builds its own adapter alongside the rows and totals readers; sharing one across all three needs a deeper refactor of the reader interface, not attempted here.

3. Best effort by contract — the service never rejects. No resolver, warehouse error, timeout, or cancelled run all degrade to coverage: 'unavailable' with a null timestamp. Per the meeting, a declared "known unknown" beats a failed run. The call is wrapped so even a resolver that throws synchronously degrades rather than escaping — QueryDataMartService relies on that to keep an orphaned promise from becoming an unhandled rejection.

4. A failed lookup never clears a persisted value. Only a resolved timestamp is written back, and ids missing from a batch response mean "no new information", not "reset".

5. Views are excluded from sources entirely. A view's lastModifiedTime tracks its definition, not its data, so surfacing it is exactly the misleading answer this feature exists to avoid. Tradeoff: if BigQuery ever stops expanding a view to its base tables, we return unavailable rather than a wrong-but-present number. That case is logged (All N referenced table(s) resolved to views) so it is observable; the fallback would be INFORMATION_SCHEMA.VIEWS.

6. Only genuine shards count in a rollup. A bare prefix LIKE would let a neighbour such as events_backup, written today, set the maximum and overstate how recent the shard set is. The suffix test mirrors GBQ_SHARDED_TABLE_SUFFIX_RE, applied to SUBSTR of the remainder so the prefix stays a bound parameter and never needs regex escaping.

7. List and canvas responses carry a slim summary without the per-table sources detail — a thousand-row page would otherwise ship megabytes of detail those surfaces never display. The full block stays on the single Data Mart GET and on refresh responses, where the tooltip renders it.

8. Naming is data_last_updated, never "freshness". A table rewritten today may have backfilled only figures from years ago. The output schema, the tool description, a per-response _instruction, and the UI copy all state that this is a storage-level write time, that null means unknown (neither fresh nor stale), and that partial reads as "at least as recent as". This is a correctness requirement for the LLM surface, not documentation polish.

9. coverage: 'partial' at the 50-table cap. BigQuery stops populating referencedTables past roughly 50 tables, and at exactly the cap a complete list is indistinguishable from a truncated one — so complete is never claimed there.

10. The metadata fan-out is capped at 10 concurrent calls, and the resolver stops between batches once its signal aborts rather than finishing work whose result has already been discarded.

11. The __TABLES__ rollup runs a tiny query in the customer's project. It scans dataset metadata only, staying inside BigQuery's minimum billing increment regardless of shard count, and registers no OWOX consumption. The alternative — MAX over the shard suffix — is wrong whenever an old shard is backfilled. Product confirmed this is acceptable.

Testing

• bigquery-source-data-last-updated.resolver.spec.ts — newest-time MAX, view drop, external-table gap, partial on a failed metadata read, both wildcard shapes (explicit events_* and an expanded shard list) collapsing to one rollup, lone date-suffixed table staying a plain table, 50-table cap, views-only case.
• source-data-last-updated.service.spec.ts — single and batch paths, missing resolver, resolver failure, synchronous resolver throw, soft-deadline expiry, abort propagation mid-flight and pre-aborted.
• refresh-data-mart-data-last-updated.service.spec.ts — one batch per storage, access filtering, definition-less drafts, unbuildable definitions dropped from the batch, partial results never clearing persisted values.
• data-mart.service.spec.ts — regression tests that both the list and canvas projections select the column. The canvas one has teeth: it fails outright if the column is missing.
• query-data-mart.service.spec.ts / query-data-mart.tool.spec.ts — composed-SQL measurement, run-record journalling, parallelism with the rows read, serialisation, calculation_origin, instruction wording, missing-block degradation.
• bigquery-api.adapter.spec.ts — shard-suffix constraint in the rollup query, identifier rejection.
• Web: data-last-updated.utils.test.ts (relative time, "Unknown" ≠ stale, ≥ for partial) plus updated canvas/list/model fixtures.
• Suites: backend src/data-marts src/ee/mcp → 5256 passed; web → 606 passed. Typecheck and lint clean on both (the repo's pre-existing backend tsc spec errors are unchanged at 127).

Note: two failures in project-list-sql-dialects.spec.ts appear only in the full parallel run and are unrelated pre-existing flakiness — that spec rebuilds TypeORM metadata from a src/**/*.entity.ts glob inside each test body with no custom timeout, so it exceeds the 5s default under load (Exceeded timeout of 5000 ms, suite wall time 141s). It passes in isolation both with and without this branch.

Deployment

One additive migration (dataLastUpdated, nullable JSON on data_mart). No new env vars, no config. The MCP response block and the run-metadata field are additive, and the run-metadata field is optional so runs recorded before this change still parse. Non-BigQuery storages report unavailable until their resolvers land. Rollback drops the column and the block.

Follow-ups

• Verify on a real BigQuery project that referencedTables expands views to base tables for our query shapes (see decision 5 — currently observable via logs).
• Report-level HTTP Data / report runs (the same service, wired into StreamHttpDataService), which is what would let reports tables show a report-level value rather than their Data Mart's.
• Remaining storages, best effort: Snowflake LAST_ALTERED, Databricks DESCRIBE DETAIL, Athena Iceberg snapshots vs. Glue metadata time; Redshift has no reliable catalog time and stays an honest null.
• Extension surface.
• Share one warehouse adapter across the rows reader, the totals reader and this lookup on the MCP path (see decision 2).

🤖 Generated with Claude Code

[prokopetsv]

Review of Data Last Updated Date (feat #6750). Summary: well-architected feature following the controller → mapper → use case → service flow, with strong test coverage and careful best-effort semantics. One blocker: the canvas sweep sends up to 1000 node ids into an endpoint that rejects more than 200, so the button always fails on large canvases. Three non-blockers inline.

[prokopetsv]

🔴 Blocker — correctness. The canvas loads up to 1000 nodes (CANVAS_DATA_MARTS_PAGE_SIZE), but this sends every visible node id in one request while the endpoint rejects >200 ids (@ArrayMaxSize(200)). On a canvas with 200+ visible marts the sweep always fails with a 400 and a generic toast. Fix: chunk ids into sequential batches of ≤200, or raise the cap to match the canvas page size.

[laskevych]

Fixed in cb29501 — real bug, thanks. Went with client-side chunking rather than raising the server cap: the endpoint keeps its bounded worst case, and the sweep now slices ids into sequential requests of ≤200 (constant documented as mirroring the DTO's ArrayMaxSize). Chunks run sequentially on purpose — each one already fans out warehouse metadata calls server-side, and overlapping sweeps would stack toward BigQuery rate limits. Results merge across chunks and apply to the react-query cache in a single write, so the flow graph still rebuilds (and re-runs fitView) once. A failed chunk no longer discards what other chunks measured; the error toast only appears when the whole sweep produced nothing.

[prokopetsv]

🟡 Non-Blocker — correctness. refreshedDataLastUpdated never resets when dataMart.id changes. If this component stays mounted across a route-param-only navigation between two Data Marts, mart A's fresh measurement renders on mart B. Neighbouring fields (contexts, availability) are re-synced via useEffect for exactly this reason. Fix: useEffect(() => setRefreshedDataLastUpdated(null), [dataMart.id]).

[laskevych]

Fixed in cb29501 exactly as suggested — a reset effect keyed by dataMart.id. Agreed on the mechanism: the derived form protects against refetch clobbering but not against identity change, and the neighbouring fields set the precedent for id-keyed sync in this component.

[prokopetsv]

🟡 Non-Blocker — readability. This describe block sits between import statements — two more imports follow after it. Hoisting makes it work, but it reads as a merge accident and import/first lint setups flag it. Move the block below all imports.

[laskevych]

Fixed in cb29501 — the describe block now sits below all imports (it was an insertion artifact; hoisting made it work, but agreed it read as a merge accident).

[prokopetsv]

🟡 Non-Blocker — docs style. House rules for docs/ cap sentences at 20 words and require active voice. Several new sentences break both, e.g. "It is measured live, alongside the data itself, every time the query runs — never cached and never billed separately…" and "Views are deliberately excluded from sources, because…". Split the long sentences and rewrite the passives.

[laskevych]

Reworked in cb29501: split the long sentences and moved the flagged passives to active voice ("Each query measures it live…", "OWOX resolves views and SQL data marts…", "sources deliberately omits views…"). One honest note: I could not find the 20-words/active-voice rule codified anywhere in the repo (checked docs/contributing/, repo root, markdownlint config) — applied it as the intended style anyway since the rewrite only improves the section. If the rule lives somewhere I missed, happy to get a pointer so future docs changes reference it.

[amarchenk0]

Please address the blocking inline comments.

[amarchenk0]

P1

dataLastUpdatedPromise is awaited together with totals after the row reader has completed. When the BigQuery dry run or metadata lookup is slow, SourceDataLastUpdatedService only resolves to unavailable at its 15-second soft timeout. An otherwise completed fast query_data_mart call therefore waits up to 15 seconds, despite this metadata being documented as best-effort and non-blocking. Do not wait for the auxiliary lookup on the response path; return the unavailable block when it has not completed by the time the query result is ready.

[laskevych]

Addressed in cb29501, with one deliberate deviation from the literal proposal. Returning unavailable the instant rows are ready would erase the block for most fast queries: the lookup starts in parallel and typically settles ~1–2s in, so a sub-second query would report unknown almost every time — hollowing out the feature's primary DoD (MCP answering "how current is this?"). Instead the finished query now grants the lookup a short grace (DEFAULT_DATA_LAST_UPDATED_GRACE_MS = 2s, ctor-injectable like the query deadline) and then degrades to unavailable. Worst-case added latency on the response path drops from 15s to 2s, while the common case keeps its value. The abandoned lookup does not keep burning warehouse calls: produce's finally aborts workController, and since this round the resolver stops between metadata batches on that signal. Covered by a regression test (stalled lookup → response returns within the grace with coverage: "unavailable"). If you feel even 2s is too much, shrinking the constant is a one-liner — but I'd argue zero-grace measurably degrades the answer quality for no real latency win.

[amarchenk0]

P1

A dry-run failure for one Data Mart rejects the entire batch. The outer service catches that rejection and returns an empty Map, so a canvas sweep does not update any healthy Data Marts on the same storage and the UI silently receives an empty items array. Isolate the failure per item and continue the batch with the results already collected.

[laskevych]

Fixed in cb29501 — agreed, this defeated the point of a sweep. resolveForSqlBatch now wraps each item in its own try/catch: a failed dry run logs a warning and skips that key, and the batch keeps measuring the rest. A missing key was already defined as "no new information" in the contract, so the healthy Data Marts update and the broken one keeps its previous persisted value — no UI-side change needed. The outer catch in SourceDataLastUpdatedService still guards whole-batch failures (adapter/credential setup). Regression test added: a two-item batch where the first dry run throws resolves the second and omits the first.