feat(sync): workspace-level git sync foundation with changeset staging#2852
feat(sync): workspace-level git sync foundation with changeset staging#2852daryllimyt wants to merge 5 commits into
Conversation
Reframe git sync from per-workflow UUID-keyed pushes to a workspace-level canonical projection addressed by stable, human-readable source IDs. - Add tracecat/workspace_sync/ package: canonical specs + deterministic serialization/hashing, workflow ORM/legacy/YAML adapter, GitHub App transport, and the projection/parse/pull/export service. - Repo layout becomes tracecat.json manifest + workflows/<slug>/definition.yml (pull dual-reads the legacy RemoteWorkflowDefinition format). - Add 6 workspace-scoped sync tables (state, resource_mapping, event, changeset, changeset_item, materialization) + migration + RLS entries. - Rewire workflow store publish/pull onto WorkspaceGitSyncService; legacy no-branch publish now synthesizes a temp branch + PR; pull gains a base-vs-local drift conflict gate. - Remove dead WorkflowSyncService and its tests; repoint store API tests. Note: changeset/event lifecycle is scaffolding only (tables written on export, no review endpoints yet); WorkspaceGitHubSyncService lacks dedicated transport tests.
daryllimyt
added
enhancement
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d92b9feb5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
✅ No security or compliance issues detected. Reviewed everything up to 911e754. Security Overview
Detected Code ChangesThe diff is too large to display a summary of code changes. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
16 issues found across 25 files
Confidence score: 2/5
- The highest-risk issue is in
tracecat/workspace_sync/service.py: pull reconciliation and pending-change staging both miss delete operations, so merges can advance base hashes while silently failing to propagate workflow deletions, leaving local/remote state inconsistent—add delete handling end-to-end in reconciliation/export and cover it with regression tests before merging. - Export flow validation is currently leaky across
tracecat/workspace_sync/schemas.py,tracecat/workspace_sync/git.py, andfrontend/src/components/organization/workspace-sync-staging.tsx, so whitespace-only messages and non-exportable selected changes can still reach Git/PR calls and fail late—enforce trimmed non-empty messages and block non-exportable items in payload construction before merge. tracecat/workspace_sync/git.pyandtracecat/workspace_sync/service.pycan fail hard on repository contents because file reads assume UTF-8 and parsing scans all files underworkflows/, so a binary/helper file can break sync unexpectedly—filter to valid workflow definition paths and make reads resilient to non-text blobs.- Input/path guardrails in
tracecat/workspace_sync/workflow.pyandtracecat/workflow/store/router.pyare weak enough to accept malformed workflow paths or empty-string branch input, which can import unexpected files or silently target the wrong base branch—tighten canonical path checks and treat empty strings as invalid explicitly before merging.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
- Drop organization_id from workspace sync tables per workspace-scoped table convention (org derives via workspace); satisfies the tenant RLS registry test introduced on main - Reparent migration onto 9b52f7f18a31 so the branch is mergeable alone - Add include_headers to webhook test fixtures for the projection carry
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 911e7542b8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| preferred_source_id = default_workflow_source_id( | ||
| alias=workflow.alias, | ||
| title=dsl.title, | ||
| ) | ||
| mapping = await self._ensure_resource_mapping( | ||
| resource_type=SyncResourceType.WORKFLOW.value, | ||
| local_id=WorkflowUUID.new(workflow.id), | ||
| preferred_source_id=preferred_source_id, | ||
| source_path=workflow_source_path(preferred_source_id), |
There was a problem hiding this comment.
When an upgraded workspace publishes a workflow before any new WorkspaceSyncResourceMapping has been created, this fallback chooses an alias/title slug as the source id, so the exporter writes workflows/<slug>/definition.yml instead of the legacy stable workflows/<wf_id>/definition.yml path that existing Git sync repos already contain. Since write_files() only adds/updates the selected files and never removes the old path, merging that publish leaves two definitions for the same workflow in the repo, and later pulls can import duplicates or stale data. Please fall back to the existing workflow-id source path or backfill/discover mappings before changing the path.
Useful? React with 👍 / 👎.
1 participant
What
Workspace-level Git sync foundation built on the projection/reconciliation model from the spec (Revamp Git Sync project):
tracecat.jsonmanifest +workflows/<source-id>/definition.ymlresource files, deterministically serialized (sorted keys,exclude_none, versionedv1:SHA-256 spec hashes). Legacywf_...repo layouts dual-read on pull.workspace_sync_resource_mappingmaps stable slug-derived source ids to workspace-local UUIDs, so one repo can hydrate multiple workspaces.clean/local_dirty/remote_ahead/diverged/conflicted) viaGET /workflows/sync/status, with pending changes fromdiff(base, P(workspace)).rendered_files) — exports always ship the reviewed snapshot, never a re-projection. Materializations record branch/commit/PR outputs.Replaces the legacy per-workflow
WorkflowSyncService(tracecat/workflow/store/sync.py); the publish path now routes through the workspace sync exporter.Database
One additive migration (
25f4e2a1c9d8): five workspace-scoped tables (workspace_sync_state,workspace_sync_resource_mapping,workspace_sync_changeset,workspace_sync_changeset_item,workspace_sync_materialization) with RLS enabled.Testing
tests/unit/test_workspace_sync.py: projection determinism, no local-UUID leakage, legacy dual-read, mapping identity, status/changeset/export flow against a mocked GitHub transport.Notes
feat/revamp-git-sync-2) land on top of this branch.Summary by cubic
Introduces workspace-level Git sync with a canonical spec, three-way status, and reviewable ChangeSets with staging, replacing the per‑workflow sync flow. Publish/pull now route through the workspace sync service, aligned with the Revamp Git Sync projection/reconciliation spec.
New Features
tracecat.json+workflows/<source-id>/definition.yml, deterministic serialization and versioned SHA‑256 spec hashes.GET /workflows/sync/status.rendered_files: select resources, then export to a branch/PR; materialization records commit/PR info.WorkflowSyncService; workflow publish/pull now usesWorkspaceGitSyncService.Migration
workspace_sync_state,workspace_sync_resource_mapping,workspace_sync_changeset,workspace_sync_changeset_item,workspace_sync_materialization,workspace_sync_event) andrendered_fileson ChangeSets; RLS enabled.organization_id), matching tenant RLS.Written for commit 911e754. Summary will update on new commits.