feat: add overview dashboard, background pipeline execution, and shared (#38)

persistence layer

  - Add repository overview page with health score ring, attention panel,
    language donut, ownership treemap, hotspots mini, decisions timeline,
    module minimap, quick actions, and active job banner
  - Extract persist_pipeline_result() into core/pipeline/persist.py so
    both CLI and server share the same persistence logic
  - Add server job_executor that runs the full pipeline in the background
    via asyncio.create_task, with batched progress writes and drain logic
  - Make sync/full-resync endpoints launch background jobs with concurrent
    run prevention (HTTP 409)
  - Reset stale running jobs on server startup (crash recovery)
  - Improve pipeline orchestrator async behavior (wrap_future, to_thread,
    periodic sleep(0) yields)
  - Update docs: ARCHITECTURE.md, CHANGELOG.md, USER_GUIDE.md,
  QUICKSTART.md

Author: swati510

docs/ARCHITECTURE.md

@@ -132,6 +132,10 @@ repowise/
 │   │   │   │       ├── fetcher.py      # EditorFileDataFetcher (DB → EditorFileData)
 │   │   │   │       ├── tech_stack.py   # filesystem tech stack + build command detection
 │   │   │   │       └── claude_md.py    # ClaudeMdGenerator subclass
+│   │   │   ├── pipeline/
+│   │   │   │   ├── orchestrator.py     # run_pipeline(), PipelineResult
+│   │   │   │   ├── persist.py          # persist_pipeline_result() — shared by CLI + server
+│   │   │   │   └── progress.py         # ProgressCallback protocol + LoggingProgressCallback
 │   │   │   ├── persistence/
 │   │   │   │   ├── models.py           # SQLAlchemy ORM models
 │   │   │   │   ├── crud.py             # async CRUD layer
@@ -165,16 +169,17 @@ repowise/
 │   │       ├── routers/         # FastAPI routers (repos, pages, jobs, symbols, graph, git, dead-code, decisions, search, claude-md)
 │   │       ├── mcp_server/      # MCP server package (8 tools, split into focused modules)
 │   │       ├── webhooks/        # GitHub + GitLab handlers
+│   │       ├── job_executor.py  # Background pipeline executor — bridges REST endpoints to core pipeline
 │   │       └── scheduler.py     # APScheduler background jobs
 │   │
 │   ├── cli/                    # Python: repowise CLI (click + rich)
 │   │   └── src/repowise/cli/
 │   │       └── commands/        # init, update, watch, serve, search, export, status, doctor, dead-code, decision, mcp, reindex, generate-claude-md
 │   │
 │   └── web/                    # Next.js 15 frontend
-│       ├── src/app/             # App Router pages (dashboard, wiki, search, graph, symbols, …)
-│       ├── src/components/      # UI primitives, layout, wiki, repos, jobs, settings
-│       └── src/lib/             # API client, SWR hooks, utilities, design tokens
+│       ├── src/app/             # App Router pages (dashboard, wiki, search, graph, symbols, overview, …)
+│       ├── src/components/      # UI primitives, layout, wiki, repos, jobs, settings, dashboard
+│       └── src/lib/             # API client, SWR hooks, utilities (health-score, format), design tokens
 │
 ├── integrations/
 │   ├── github-action/           # action.yml + Dockerfile entrypoint
@@ -613,6 +618,46 @@ are skipped, and generation continues from the last checkpoint.
 `repowise init` is fully idempotent. Running it twice produces the same result.
 Running it after a partial previous run completes only the remaining pages.
 
+### 5.7 Shared Persistence (`pipeline/persist.py`)
+
+The persistence logic for storing a `PipelineResult` into the database (graph nodes,
+edges, symbols, pages, git metadata, dead code findings, decision records) was
+extracted from the CLI's `init_cmd.py` into `core/pipeline/persist.py`. Both the
+CLI and the server's background job executor call `persist_pipeline_result()` — zero
+duplication.
+
+FTS indexing is intentionally excluded from this function. Callers must run it
+separately after the session closes to avoid SQLite write-lock conflicts.
+
+### 5.8 Background Job Executor (`server/job_executor.py`)
+
+The server can now run full pipeline jobs in the background, triggered by the
+`POST /api/repos/{id}/sync` and `POST /api/repos/{id}/full-resync` endpoints.
+
+`execute_job()` is the single entry point, launched via `asyncio.create_task()`:
+
+1. Marks the job as `running`
+2. Resolves the LLM provider from server config
+3. Runs `run_pipeline()` with a `JobProgressCallback` that writes progress to the
+   `GenerationJob` table (the SSE stream endpoint polls this table)
+4. Persists results via `persist_pipeline_result()`
+5. Marks the job as `completed` (or `failed` on error)
+
+Progress updates are batched (every 5 items) to avoid per-item DB overhead. Before
+writing the final job status, all in-flight progress tasks are drained to prevent a
+late `running` update from overwriting `completed`.
+
+Concurrent pipeline runs on the same repository are prevented at the endpoint level
+(returns HTTP 409 if a pending/running job already exists).
+
+### 5.9 Async Pipeline Improvements
+
+The pipeline orchestrator now keeps the event loop responsive during CPU-bound work:
+- File I/O uses `asyncio.wrap_future()` instead of blocking `as_completed()`
+- Graph building runs in a thread via `asyncio.to_thread()`
+- The parse loop yields control every 50 files with `asyncio.sleep(0)`
+- Thread pool shutdown is non-blocking via `asyncio.to_thread()`
+
 ---
 
 ## 6. Maintenance Path — Keeping Docs in Sync
@@ -983,7 +1028,7 @@ and Cline. This config is printed at the end of `repowise init`.
 Served on port `7337` alongside the web UI. All endpoints are prefixed with `/api/`.
 
 Key routers:
-- `/api/repos` — register repos, trigger sync, full-resync
+- `/api/repos` — register repos, trigger sync, full-resync (now launches background pipeline jobs with concurrent-run prevention)
 - `/api/pages` — read pages, version history, force-regenerate single page
 - `/api/search` — semantic (LanceDB or pgvector) and full-text (SQLite FTS5 / PostgreSQL tsvector) search
 - `/api/jobs` — job status, SSE stream for live progress updates
@@ -1004,6 +1049,12 @@ Key routers:
 - `/health` — liveness + readiness (checks DB + provider)
 - `/metrics` — Prometheus-compatible metrics (job counts, token totals, stale count)
 
+**Server lifecycle:**
+- On startup, any jobs left in `running` state from a previous server instance are
+  automatically reset to `failed` (crash recovery).
+- Background pipeline tasks are tracked in `app.state.background_tasks` to prevent
+  garbage collection of `asyncio.Task` references.
+
 Authentication is optional. Set `REPOWISE_API_KEY` to require bearer token auth on
 all non-`/health` endpoints. Default (no key set): fully open, suitable for local use.
 
@@ -1014,7 +1065,8 @@ Served from the same port as the API. All routes under `/`:
 | Route | Content |
 |-------|---------|
 | `/` | Dashboard: all repos, recent jobs, stale page counts, token usage |
-| `/repos/[id]` | Repo overview wiki page + file tree sidebar |
+| `/repos/[id]` | Repo layout with file tree sidebar |
+| `/repos/[id]/overview` | **Overview dashboard** — health score ring, attention panel, language donut, ownership treemap, hotspots mini, decisions timeline, module minimap, quick actions, active job banner |
 | `/repos/[id]/wiki/[...slug]` | Individual wiki page with MDX rendering |
 | `/repos/[id]/search` | Semantic search results |
 | `/repos/[id]/graph` | D3 force-directed dependency graph |
@@ -1023,6 +1075,8 @@ Served from the same port as the API. All routes under `/`:
 | `/repos/[id]/ownership` | Ownership treemap — files colored by primary owner, sized by LOC |
 | `/repos/[id]/hotspots` | Hotspot list — top 20 files with churn + complexity bars |
 | `/repos/[id]/dead-code` | Dead code report — three tabs: Files, Exports, Internals |
+| `/repos/[id]/decisions` | Architectural decision records |
+| `/repos/[id]/chat` | Codebase chat with streaming LLM responses |
 | `/settings` | Provider config, polling interval, cascade budget |
 
 **Key rendering behavior:**
@@ -1201,6 +1255,33 @@ repos.last_sync_commit = HEAD
 state.json updated
 ```
 
+### Server-triggered pipeline flow
+
+```
+Web UI "Sync" / "Full Re-index" button
+    │
+    ▼
+POST /api/repos/{id}/sync (or /full-resync)
+    │
+    ├── Check: no pending/running job for this repo (else → 409)
+    ▼
+Create GenerationJob (status=pending, commit)
+    │
+    ▼
+asyncio.create_task(execute_job(job_id, app_state))
+    │  (strong ref in app.state.background_tasks)
+    ▼
+execute_job():
+    ├── Mark job "running"
+    ├── Resolve LLM provider from server config
+    ├── run_pipeline() with JobProgressCallback
+    │       └── writes progress to GenerationJob table every 5 items
+    ├── persist_pipeline_result() (shared with CLI)
+    ├── FTS index new pages
+    ├── drain_and_stop() progress tasks
+    └── Mark job "completed" (or "failed" on error)
+```
+
 ### MCP query flow
 
 ```

docs/CHANGELOG.md

@@ -12,6 +12,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- **Overview dashboard** (`/repos/[id]/overview`) — new landing page for each repository with:
+  - Health score ring (composite of doc coverage, freshness, dead code, hotspot density, silo risk)
+  - Attention panel highlighting items needing action (stale docs, high-risk hotspots, dead code)
+  - Language donut chart, ownership treemap, hotspots mini-list
+  - Decisions timeline, module minimap (interactive graph summary)
+  - Quick actions panel (sync, full re-index, generate CLAUDE.md, export)
+  - Active job banner with live progress polling
+- **Background pipeline execution** — `POST /api/repos/{id}/sync` and `POST /api/repos/{id}/full-resync` now launch the full pipeline in the background instead of only creating a pending job. Concurrent runs on the same repo return HTTP 409.
+- **Shared persistence layer** (`core/pipeline/persist.py`) — `persist_pipeline_result()` extracted from CLI, reused by both CLI and server job executor
+- **Job executor** (`server/job_executor.py`) — background task that runs `run_pipeline()`, writes progress to the `GenerationJob` table, and persists all results
+- **Server crash recovery** — stale `running` jobs are reset to `failed` on server startup
+- **Async pipeline improvements** — `asyncio.wrap_future` for file I/O, `asyncio.to_thread` for graph building and thread pool shutdown, periodic `asyncio.sleep(0)` yields during parsing
+- **Health score utility** (`web/src/lib/utils/health-score.ts`) — composite health score computation, attention item builder, and language aggregation for the overview dashboard
+
+### Changed
+- `init_cmd.py` refactored to use shared `persist_pipeline_result()` instead of inline persistence logic
+- Pipeline orchestrator uses async-friendly patterns to keep the event loop responsive during ingestion
+- Sidebar and mobile nav updated to include "Overview" link
+
 - Monorepo scaffold: uv workspace with `packages/core`, `packages/cli`, `packages/server`, `packages/web`
 - Provider abstraction layer: `BaseProvider`, `GeneratedResponse`, `ProviderError`, `RateLimitError`
 - `AnthropicProvider` with prompt caching support

docs/QUICKSTART.md

@@ -91,7 +91,7 @@ repowise watch
 
 ## Web UI
 
-Repowise includes a full web dashboard with a wiki browser, interactive dependency graph, codebase chat, search, code ownership, hotspots, and dead code detection.
+Repowise includes a full web dashboard with a repository overview, wiki browser, interactive dependency graph, codebase chat, search, code ownership, hotspots, and dead code detection. The overview page shows a health score, attention items, language breakdown, ownership treemap, and quick actions.
 
 ### With Node.js installed
 

docs/USER_GUIDE.md

@@ -616,6 +616,20 @@ Open **http://localhost:3000** in your browser.
 **Dashboard** (`/`)
 Home page with aggregate stats (total pages, fresh/stale counts, dead code findings), a list of indexed repositories, and recent job status.
 
+**Repository Overview** (`/repos/[id]/overview`)
+A single-page dashboard for each repository that aggregates key health signals. Includes:
+- **Health score ring** — composite score (0–100) computed from documentation coverage, freshness, dead code ratio, hotspot density, and ownership silo risk
+- **Attention panel** — prioritized list of items needing action (stale docs, high-churn hotspots, dead code findings)
+- **Language donut** — breakdown of codebase by programming language
+- **Ownership treemap** — visualizes code ownership distribution across modules
+- **Hotspots mini** — top high-churn files at a glance
+- **Decisions timeline** — recent architectural decisions
+- **Module minimap** — compact interactive graph of module relationships
+- **Quick actions** — one-click buttons for sync, full re-index, CLAUDE.md generation, and export
+- **Active job banner** — shows progress of running pipeline jobs with live polling
+
+The overview page degrades gracefully — each data section loads independently, so partial data (e.g., missing git metadata) still renders a useful dashboard.
+
 **Wiki Browser** (`/repos/[id]/wiki/...`)
 The heart of repowise. Browse AI-generated documentation for every file and module. Each page includes:
 - Rendered markdown with syntax-highlighted code blocks and Mermaid diagrams

packages/cli/src/repowise/cli/commands/init_cmd.py

@@ -134,113 +134,35 @@ async def _persist_result(
     """
     from repowise.cli.helpers import get_db_url_for_repo
     from repowise.core.persistence import (
-        batch_upsert_graph_edges,
-        batch_upsert_graph_nodes,
-        batch_upsert_symbols,
+        FullTextSearch,
         create_engine,
         create_session_factory,
         get_session,
         init_db,
         upsert_repository,
     )
-    from repowise.core.persistence.crud import (
-        save_dead_code_findings,
-        upsert_git_metadata_bulk,
-    )
+    from repowise.core.pipeline import persist_pipeline_result
 
     url = get_db_url_for_repo(repo_path)
     engine = create_engine(url)
     await init_db(engine)
     sf = create_session_factory(engine)
 
+    fts = None
+    if result.generated_pages:
+        fts = FullTextSearch(engine)
+        await fts.ensure_index()
+
     async with get_session(sf) as session:
         repo = await upsert_repository(
             session,
             name=result.repo_name,
             local_path=str(repo_path),
         )
-        repo_id = repo.id
-
-        # Pages (if generated)
-        if result.generated_pages:
-            from repowise.core.persistence import upsert_page_from_generated
-
-            for page in result.generated_pages:
-                await upsert_page_from_generated(session, page, repo_id)
-
-        # Graph nodes
-        graph = result.graph_builder.graph()
-        pr = result.graph_builder.pagerank()
-        bc = result.graph_builder.betweenness_centrality()
-        cd = result.graph_builder.community_detection()
-        nodes = []
-        for node_path in graph.nodes:
-            data = graph.nodes[node_path]
-            nodes.append(
-                {
-                    "node_id": node_path,
-                    "symbol_count": data.get("symbol_count", 0),
-                    "has_error": data.get("has_error", False),
-                    "is_test": data.get("is_test", False),
-                    "is_entry_point": data.get("is_entry_point", False),
-                    "language": data.get("language", "unknown"),
-                    "pagerank": pr.get(node_path, 0.0),
-                    "betweenness": bc.get(node_path, 0.0),
-                    "community_id": cd.get(node_path, 0),
-                }
-            )
-        if nodes:
-            await batch_upsert_graph_nodes(session, repo_id, nodes)
-
-        # Graph edges
-        edges = []
-        for u, v, data in graph.edges(data=True):
-            edges.append(
-                {
-                    "source_node_id": u,
-                    "target_node_id": v,
-                    "imported_names_json": json.dumps(data.get("imported_names", [])),
-                    "edge_type": data.get("edge_type", "imports"),
-                }
-            )
-        if edges:
-            await batch_upsert_graph_edges(session, repo_id, edges)
-
-        # Symbols
-        all_symbols = []
-        for pf in result.parsed_files:
-            for sym in pf.symbols:
-                sym.file_path = pf.file_info.path
-                all_symbols.append(sym)
-        if all_symbols:
-            await batch_upsert_symbols(session, repo_id, all_symbols)
-
-        # Git metadata
-        if result.git_metadata_list:
-            await upsert_git_metadata_bulk(session, repo_id, result.git_metadata_list)
-
-        # Dead code findings
-        if result.dead_code_report and result.dead_code_report.findings:
-            await save_dead_code_findings(session, repo_id, result.dead_code_report.findings)
-
-        # Decision records
-        if result.decision_report and result.decision_report.decisions:
-            import dataclasses as _dc
-
-            from repowise.core.persistence.crud import bulk_upsert_decisions
-
-            await bulk_upsert_decisions(
-                session,
-                repo_id,
-                [_dc.asdict(d) for d in result.decision_report.decisions],
-            )
-
-    # FTS indexing (only when pages were generated)
-    if result.generated_pages:
-        from repowise.core.persistence import FullTextSearch
+        await persist_pipeline_result(result, session, repo.id)
 
-        fts = FullTextSearch(engine)
-        await fts.ensure_index()
+    # FTS indexing is done outside the session to avoid SQLite write conflicts
+    if fts is not None and result.generated_pages:
         for page in result.generated_pages:
             await fts.index(page.page_id, page.title, page.content)