feat: Real-time query observability dashboard

[jack-arturo]

Problem

Currently, the only visibility into AutoMem's behavior is through raw server logs on the agent end. There's no insight into what's happening on the server side during memory operations.

Proposed Solution

Add a real-time observability layer that exposes query metrics and traversal details. This could be:

Implementation Options

1. Terminal-based - Simple CLI tool that tails structured logs
2. WebSocket-based - Real-time web dashboard (could integrate with existing graph-viewer)
3. Both - Terminal for quick debugging, web for richer visualization

Metrics to Expose

Query Metrics

• Incoming queries (recall, store, associate, etc.)
• Query parameters and filters used
• Response times

Graph Traversal Stats

• Number of nodes traversed
• Number of hops taken
• Relationship types followed
• Duplicate detection counts

Quality Metrics (if low-cost)

• Query optimization score - how well-formed was the agent's query?
• Result relevancy estimate - how relevant were the returned memories?
• Cache hit rates (if applicable)

Enrichment Pipeline

• Queue depth
• Processing times
• Entity extraction results

Prior Art

The existing /enrichment/status and /analyze endpoints provide some static metrics. This would extend that to real-time streaming.

Notes

• Should have minimal performance overhead when not actively monitored
• Consider structured logging (JSON) for easy parsing
• Could leverage existing FalkorDB Browser infrastructure (port 3001)

---

Deferred until current branch is merged

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Approach

Ticket Summary

Add a real-time observability system that streams query metrics, graph traversal statistics, and enrichment pipeline status to both terminal CLI and web dashboard consumers, with minimal performance overhead when not monitored.

Observations

Codebase Summary

AutoMem is a Flask-based memory management API organized into blueprints (health, memory, recall, enrichment, consolidation, graph, admin) with 26 endpoints that interact with FalkorDB (graph database) and Qdrant (vector database). The codebase uses standard Python logging, manual timing with time.perf_counter(), and has an established pattern of separate optional services (like FalkorDB Browser on port 3001 and mcp-sse-server for MCP protocol). There are no WebSocket libraries installed, but a separate Node.js service demonstrates successful SSE (Server-Sent Events) implementation patterns including session management and heartbeats.

Assumptions

Assumption 1: Real-time streaming protocol selection

Options Considered:

• Use Flask-SocketIO for WebSocket-based bidirectional streaming (requires new heavyweight dependencies)
• Use SSE (Server-Sent Events) for server-to-client streaming (pure HTTP, aligns with existing mcp-sse-server patterns)

Chosen Option: SSE-based streaming

Rationale: This aligns with existing infrastructure patterns, requires no new major dependencies, provides sufficient functionality for metrics streaming (which is inherently server-to-client), and matches the lightweight approach demonstrated in mcp-sse-server. The ticket's requirements don't genuinely require bidirectional communication for metrics consumption.

Assumption 2: FalkorDB traversal metrics collection approach

Options Considered:

• Use GRAPH.PROFILE command for execution plans with per-operation metrics (accurate but adds overhead)
• Parse Cypher queries to estimate traversal complexity (lightweight but less accurate)
• Track only response times and rely on FalkorDB's built-in stats (simplest, minimal overhead)

Chosen Option: Hybrid approach: Track basic stats for all queries, but optionally use GRAPH.PROFILE for complex graph operations in specific endpoints (recall, graph traversal, analytics) when detailed metrics would be valuable

Rationale: This provides detailed insights where they matter most while keeping overhead minimal for simple CRUD operations. Profile data collection will be conditional based on whether metrics consumers are connected.

Assumption 3: Metrics storage and retention strategy

Options Considered:

• In-memory ring buffer (lightweight, no persistence, lost on restart)
• Structured logging to files with external aggregation (requires external tools)
• Database persistence (adds complexity and storage requirements)

Chosen Option: In-memory ring buffer with configurable size (default 1000 events)

Rationale: This provides recent history for connected clients without persistence overhead, aligns with the "minimal overhead when not monitored" requirement, and keeps the implementation self-contained. Historical analysis can be added later through structured logging if needed.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Core Metrics Collection Infrastructure

Create the foundational metrics collection layer that captures operational data from API endpoints with minimal overhead and zero impact when no clients are monitoring.

Task 1: Create Metrics Collector Service

Implement a centralized metrics collection service in a new automem/observability/ module that manages metric events and connected client sessions.

• Create automem/observability/collector.py with a MetricsCollector class that maintains an in-memory ring buffer (configurable size, default 1000 events) for recent metrics
• Implement thread-safe operations using locks for concurrent access from multiple request threads
• Add methods for emitting different metric types (query_started, query_completed, enrichment_event, graph_traversal)
• Track active client connections with a session registry and only collect detailed metrics when clients are connected (overhead optimization)
• Provide a singleton instance accessible across the application for consistent metric emission

Task 2: Add Endpoint Instrumentation Decorator

Create a decorator that wraps API endpoints to automatically capture request/response metrics without modifying existing endpoint logic.

• Create automem/observability/instrumentation.py with an @observe_endpoint decorator
• Capture incoming request metadata (method, path, query parameters, request headers for API token validation status)
• Measure endpoint execution time using time.perf_counter() (consistent with existing timing pattern)
• Record response metadata (status code, response size, query_time_ms from response JSON if present)
• Emit metrics to the collector only when monitoring clients are active
• Handle exceptions gracefully and record error metrics without disrupting endpoint behavior

Task 3: Instrument Key API Endpoints

Apply the instrumentation decorator to all endpoints identified in the exploration across the seven blueprint files.

• Update endpoints in automem/api/memory.py (store, update, delete, by-tag, associate) to use @observe_endpoint
• Update endpoints in automem/api/recall.py (recall, startup-recall, analyze, related) to use @observe_endpoint
• Update endpoints in automem/api/graph.py (snapshot, neighbors, stats, types, relations) to use @observe_endpoint
• Update endpoints in automem/api/enrichment.py (status, reprocess) to use @observe_endpoint
• Update endpoints in automem/api/consolidation.py (consolidate, status) to use @observe_endpoint
• Update endpoints in automem/api/admin.py (reembed, sync) to use @observe_endpoint
• Configure decorator to extract and include endpoint-specific context (e.g., memory_id from path, query filters from parameters)

🤖 Prompt for AI agents

Create foundational metrics collection infrastructure that captures API endpoint
metrics with minimal overhead.

Create `automem/observability/collector.py`:
- Implement MetricsCollector class with thread-safe in-memory ring buffer
(default 1000 events)
- Add session registry to track active monitoring clients
- Implement methods for emitting metric types: query_started, query_completed,
enrichment_event, graph_traversal
- Only collect detailed metrics when clients are connected
- Provide singleton instance

Create `automem/observability/instrumentation.py`:
- Implement `@observe_endpoint` decorator
- Capture request metadata (method, path, query params, headers)
- Measure execution time with `time.perf_counter()`
- Record response metadata (status, size, query_time_ms)
- Emit metrics only when monitoring clients active
- Handle exceptions without disrupting endpoint behavior

Apply decorator to endpoints:
- Apply `@observe_endpoint` to all endpoints in: memory.py, recall.py, graph.py,
enrichment.py, consolidation.py, admin.py blueprints
- Extract endpoint-specific context (memory_id, query filters) in decorator
configuration

Phase 2: SSE Streaming Infrastructure

Implement Server-Sent Events endpoints and session management to enable real-time metric streaming to connected clients with heartbeat handling.

Task 1: Create Metrics Streaming Blueprint

Add a new blueprint for observability endpoints that handles SSE connections and metric delivery.

• Create automem/api/metrics.py blueprint with two routes: GET /metrics/stream (SSE) and GET /metrics/history (JSON snapshot)
• Implement SSE stream using Flask's Response with mimetype='text/event-stream'
• Set appropriate headers (Cache-Control: no-cache, X-Accel-Buffering: no, Connection: keep-alive) following mcp-sse-server patterns
• Register session with MetricsCollector when SSE connection established
• Yield recent buffer history on initial connection, then stream new events as they occur
• Clean up session on connection close

Task 2: Add Session Management and Heartbeat

Implement robust session lifecycle handling with heartbeat mechanism to prevent connection timeouts in proxy environments.

• Add session tracking to MetricsCollector with connection metadata (client ID, connection time, last heartbeat)
• Implement 20-second heartbeat interval (matching mcp-sse-server pattern) that sends : ping\n\n comments
• Use Flask's @stream_with_context to properly handle streaming context and cleanup
• Register cleanup handlers for connection close events that remove sessions and stop heartbeats
• Notify MetricsCollector when last client disconnects to stop detailed metric collection

Task 3: Register Metrics Blueprint

Integrate the new observability endpoints into the main Flask application alongside existing blueprints.

• Import metrics blueprint in app.py
• Register blueprint with URL prefix /metrics following existing pattern for health, memory, recall blueprints
• Ensure API token middleware exempts /metrics/stream and /metrics/history similar to /health exemption (or require token based on security preference)
• Add basic documentation comment in app.py describing the metrics endpoints

🤖 Prompt for AI agents

Implement SSE streaming infrastructure for real-time metrics delivery.

Create `automem/api/metrics.py` blueprint:
- Add route `GET /metrics/stream` returning SSE stream (Flask Response with
mimetype='text/event-stream')
- Add route `GET /metrics/history` returning JSON snapshot of recent metrics
- Set SSE headers: Cache-Control: no-cache, X-Accel-Buffering: no, Connection:
keep-alive
- Register session with MetricsCollector on connection
- Yield buffer history on initial connection, then stream new events
- Clean up session on close

Add session management to MetricsCollector:
- Track sessions with metadata (client ID, connection time, last heartbeat)
- Implement 20-second heartbeat sending `: ping\n\n`
- Use `@stream_with_context` for proper cleanup
- Remove sessions on disconnect
- Stop detailed collection when last client disconnects

Register blueprint in `app.py`:
- Import and register metrics blueprint with `/metrics` prefix
- Exempt `/metrics/stream` and `/metrics/history` from API token middleware (or
enforce based on security needs)
- Add documentation comment

Phase 3: FalkorDB Query Profiling and Traversal Metrics

Extend instrumentation to capture graph traversal statistics using FalkorDB's profiling capabilities for graph-intensive operations.

Task 1: Create FalkorDB Query Profiler

Implement a query profiling wrapper that extracts traversal metrics from FalkorDB execution plans.

• Create automem/observability/graph_profiler.py with functions to execute GRAPH.PROFILE for query analysis
• Parse execution plan results to extract traversal statistics (operation counts, records produced, operation runtime)
• Calculate aggregate metrics: total nodes scanned, total relationships traversed, hop count estimates
• Return structured profile data including per-operation breakdown and aggregate traversal metrics
• Handle profiling errors gracefully and fall back to basic timing when profiling fails

Task 2: Add Conditional Profiling to Graph-Intensive Endpoints

Selectively enable profiling for endpoints that perform significant graph traversal when monitoring is active.

• Update automem/api/recall.py recall endpoint to conditionally profile graph expansion queries when metrics clients connected
• Update automem/api/graph.py snapshot and neighbors endpoints to profile traversal queries
• Update automem/api/recall.py analyze endpoint to profile analytics aggregation queries
• Emit graph_traversal metrics with detailed stats (nodes_traversed, relationships_followed, hops_taken, duplicates_detected)
• Skip profiling when no metrics clients are connected to avoid overhead

Task 3: Extract Traversal Context from Query Results

Augment existing query result processing to capture traversal metadata available in standard QueryResult objects.

• Update query result handling in instrumented endpoints to extract stats array from FalkorDB QueryResult
• Parse stats for execution time and mutation counts (nodes created/deleted, relationships created/deleted)
• Include available stats in query_completed metrics even when profiling is disabled
• Track Cypher query patterns and classify by complexity (simple MATCH, multi-hop traversal, aggregation)

🤖 Prompt for AI agents

Extend instrumentation to capture FalkorDB graph traversal statistics.

Create `automem/observability/graph_profiler.py`:
- Implement functions to execute GRAPH.PROFILE for queries
- Parse execution plans to extract traversal stats (operation counts, records
produced, runtime)
- Calculate aggregate metrics: nodes scanned, relationships traversed, hop count
estimates
- Return structured profile data with per-operation breakdown
- Handle errors gracefully with fallback to basic timing

Add conditional profiling to graph endpoints:
- Update recall.py recall endpoint to conditionally profile graph expansion when
clients connected
- Update graph.py snapshot and neighbors endpoints to profile traversal queries
- Update recall.py analyze endpoint to profile analytics queries
- Emit graph_traversal metrics with detailed stats (nodes_traversed,
relationships_followed, hops_taken, duplicates_detected)
- Skip profiling when no clients connected

Extract traversal context from query results:
- Extract stats array from FalkorDB QueryResult in instrumented endpoints
- Parse execution time and mutation counts
- Include stats in query_completed metrics even without profiling
- Classify Cypher query patterns by complexity

Phase 4: Enrichment Pipeline Event Integration

Hook into existing enrichment infrastructure to emit real-time events about queue status and processing completion.

Task 1: Instrument Enrichment Queue Operations

Add metric emission to enrichment queue management for queue depth and state transitions.

• Locate enrichment queue operations in the codebase (likely in background worker threads)
• Emit metrics when items are enqueued (memory ID, queue size after enqueue)
• Emit metrics when items transition to inflight state (memory ID, queue size, pending count, inflight count)
• Emit metrics when items complete processing (memory ID, success/failure, processing time, entities extracted)
• Include enrichment statistics from state.enrichment_stats in completion events

Task 2: Add Enrichment Event Types to Metrics Schema

Define structured event types for enrichment pipeline events in the metrics collector.

• Add enrichment_queued event type with fields (memory_id, queue_depth, timestamp)
• Add enrichment_processing event type with fields (memory_id, queue_depth, pending, inflight)
• Add enrichment_completed event type with fields (memory_id, duration_ms, success, entities_extracted, enrichment_status)
• Add enrichment_failed event type with fields (memory_id, duration_ms, error_type, attempt_number)

Task 3: Expose Enrichment Metrics in Stream

Ensure enrichment events are included in the SSE stream alongside query metrics.

• Configure MetricsCollector to accept enrichment events from background threads
• Include enrichment events in the SSE stream with appropriate event type labels
• Add enrichment summary to /metrics/history JSON endpoint showing current queue state and recent processing stats

🤖 Prompt for AI agents

Integrate enrichment pipeline events into metrics collection.

Instrument enrichment queue operations:
- Locate enrichment queue operations in the codebase
- Emit metrics on enqueue (memory_id, queue_size)
- Emit metrics on inflight transition (memory_id, queue_size, pending, inflight)
- Emit metrics on completion (memory_id, success/failure, processing_time,
entities_extracted)
- Include enrichment_stats in completion events

Define enrichment event types in MetricsCollector:
- Add enrichment_queued type with fields: memory_id, queue_depth, timestamp
- Add enrichment_processing type with fields: memory_id, queue_depth, pending,
inflight
- Add enrichment_completed type with fields: memory_id, duration_ms, success,
entities_extracted, enrichment_status
- Add enrichment_failed type with fields: memory_id, duration_ms, error_type,
attempt_number

Expose enrichment metrics:
- Configure MetricsCollector to accept events from background threads
- Include enrichment events in SSE stream with event type labels
- Add enrichment summary to `/metrics/history` showing queue state and
processing stats

Phase 5: Client Interfaces - Dashboard and CLI

Build web dashboard and terminal CLI for consuming metrics streams, leveraging existing static serving patterns and Python scripting.

Task 1: Create Web Dashboard Static Files

Build a simple HTML/JavaScript dashboard that connects to the SSE endpoint and visualizes metrics in real-time.

• Create automem/static/dashboard.html with embedded JavaScript (no build system needed for simplicity)
• Implement SSE EventSource connection to /metrics/stream with reconnection logic
• Display real-time metrics in sections: Active Queries, Recent Completions, Graph Traversal Stats, Enrichment Queue
• Show query response time histogram or timeline chart using Chart.js or vanilla canvas
• Add filters for viewing specific endpoint types or time ranges (client-side filtering of buffered events)
• Include connection status indicator and manual refresh button

Task 2: Add Static File Serving Route

Serve the dashboard HTML from a Flask route for easy access during development and optional production use.

• Add route GET /dashboard to metrics blueprint that serves dashboard.html
• Configure Flask to serve static files from automem/static/ directory if not already configured
• Add CORS headers if needed for local development scenarios
• Document how to access dashboard in README (e.g., http://localhost:8080/dashboard)

Task 3: Create Terminal CLI Script

Build a Python CLI script that tails the metrics stream and displays formatted output for terminal-based monitoring.

• Create scripts/metrics_cli.py that connects to /metrics/stream endpoint using requests with stream=True
• Parse SSE events and format for terminal display with color coding (rich library optional, or ANSI codes)
• Display real-time log of queries with response times, enrichment events, and errors
• Add command-line arguments for filtering (--endpoint-type, --min-duration, --show-enrichment)
• Include connection retry logic with exponential backoff
• Add usage documentation in docs/METRICS_CLI.md

Task 4: Documentation and Configuration

Document the observability system and provide configuration options for tuning behavior.

• Create docs/OBSERVABILITY.md with overview, architecture diagram, and usage examples
• Document SSE endpoint API contract (event types, payload schemas)
• Add environment variable for configuring buffer size (METRICS_BUFFER_SIZE, default 1000)
• Add environment variable for enabling/disabling profiling (METRICS_ENABLE_PROFILING, default false)
• Document how to deploy dashboard in production environments (optional service pattern like FalkorDB Browser)
• Add troubleshooting section for common issues (connection timeouts, missing metrics, high overhead)

🤖 Prompt for AI agents

Build client interfaces for consuming metrics: web dashboard and terminal CLI.

Create web dashboard:
- Create `automem/static/dashboard.html` with embedded JavaScript
- Implement SSE EventSource connecting to `/metrics/stream` with reconnection
- Display sections: Active Queries, Recent Completions, Graph Traversal Stats,
Enrichment Queue
- Add query response time visualization (Chart.js or vanilla canvas)
- Add client-side filters for endpoint types and time ranges
- Include connection status indicator and manual refresh

Add dashboard serving:
- Add route `GET /dashboard` to metrics blueprint serving dashboard.html
- Configure Flask to serve static files from `automem/static/`
- Add CORS headers if needed
- Document dashboard access in README

Create terminal CLI:
- Create `scripts/metrics_cli.py` connecting to `/metrics/stream` with requests
stream=True
- Parse and format SSE events for terminal with color coding
- Display real-time log of queries, enrichment events, errors
- Add CLI arguments: --endpoint-type, --min-duration, --show-enrichment
- Implement connection retry with exponential backoff
- Document usage in `docs/METRICS_CLI.md`

Add documentation:
- Create `docs/OBSERVABILITY.md` with overview, architecture, usage examples
- Document SSE endpoint API contract (event types, schemas)
- Add environment variables: METRICS_BUFFER_SIZE (default 1000),
METRICS_ENABLE_PROFILING (default false)
- Document production deployment patterns
- Add troubleshooting section

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Create foundational metrics collection infrastructure that captures API endpoint
metrics with minimal overhead.

Create `automem/observability/collector.py`:
- Implement MetricsCollector class with thread-safe in-memory ring buffer
(default 1000 events)
- Add session registry to track active monitoring clients
- Implement methods for emitting metric types: query_started, query_completed,
enrichment_event, graph_traversal
- Only collect detailed metrics when clients are connected
- Provide singleton instance

Create `automem/observability/instrumentation.py`:
- Implement `@observe_endpoint` decorator
- Capture request metadata (method, path, query params, headers)
- Measure execution time with `time.perf_counter()`
- Record response metadata (status, size, query_time_ms)
- Emit metrics only when monitoring clients active
- Handle exceptions without disrupting endpoint behavior

Apply decorator to endpoints:
- Apply `@observe_endpoint` to all endpoints in: memory.py, recall.py, graph.py,
enrichment.py, consolidation.py, admin.py blueprints
- Extract endpoint-specific context (memory_id, query filters) in decorator
configuration
===============================================================================

Task: 2

Implement SSE streaming infrastructure for real-time metrics delivery.

Create `automem/api/metrics.py` blueprint:
- Add route `GET /metrics/stream` returning SSE stream (Flask Response with
mimetype='text/event-stream')
- Add route `GET /metrics/history` returning JSON snapshot of recent metrics
- Set SSE headers: Cache-Control: no-cache, X-Accel-Buffering: no, Connection:
keep-alive
- Register session with MetricsCollector on connection
- Yield buffer history on initial connection, then stream new events
- Clean up session on close

Add session management to MetricsCollector:
- Track sessions with metadata (client ID, connection time, last heartbeat)
- Implement 20-second heartbeat sending `: ping\n\n`
- Use `@stream_with_context` for proper cleanup
- Remove sessions on disconnect
- Stop detailed collection when last client disconnects

Register blueprint in `app.py`:
- Import and register metrics blueprint with `/metrics` prefix
- Exempt `/metrics/stream` and `/metrics/history` from API token middleware (or
enforce based on security needs)
- Add documentation comment
===============================================================================

Task: 3

Extend instrumentation to capture FalkorDB graph traversal statistics.

Create `automem/observability/graph_profiler.py`:
- Implement functions to execute GRAPH.PROFILE for queries
- Parse execution plans to extract traversal stats (operation counts, records
produced, runtime)
- Calculate aggregate metrics: nodes scanned, relationships traversed, hop count
estimates
- Return structured profile data with per-operation breakdown
- Handle errors gracefully with fallback to basic timing

Add conditional profiling to graph endpoints:
- Update recall.py recall endpoint to conditionally profile graph expansion when
clients connected
- Update graph.py snapshot and neighbors endpoints to profile traversal queries
- Update recall.py analyze endpoint to profile analytics queries
- Emit graph_traversal metrics with detailed stats (nodes_traversed,
relationships_followed, hops_taken, duplicates_detected)
- Skip profiling when no clients connected

Extract traversal context from query results:
- Extract stats array from FalkorDB QueryResult in instrumented endpoints
- Parse execution time and mutation counts
- Include stats in query_completed metrics even without profiling
- Classify Cypher query patterns by complexity
===============================================================================

Task: 4

Integrate enrichment pipeline events into metrics collection.

Instrument enrichment queue operations:
- Locate enrichment queue operations in the codebase
- Emit metrics on enqueue (memory_id, queue_size)
- Emit metrics on inflight transition (memory_id, queue_size, pending, inflight)
- Emit metrics on completion (memory_id, success/failure, processing_time,
entities_extracted)
- Include enrichment_stats in completion events

Define enrichment event types in MetricsCollector:
- Add enrichment_queued type with fields: memory_id, queue_depth, timestamp
- Add enrichment_processing type with fields: memory_id, queue_depth, pending,
inflight
- Add enrichment_completed type with fields: memory_id, duration_ms, success,
entities_extracted, enrichment_status
- Add enrichment_failed type with fields: memory_id, duration_ms, error_type,
attempt_number

Expose enrichment metrics:
- Configure MetricsCollector to accept events from background threads
- Include enrichment events in SSE stream with event type labels
- Add enrichment summary to `/metrics/history` showing queue state and
processing stats
===============================================================================

Task: 5

Build client interfaces for consuming metrics: web dashboard and terminal CLI.

Create web dashboard:
- Create `automem/static/dashboard.html` with embedded JavaScript
- Implement SSE EventSource connecting to `/metrics/stream` with reconnection
- Display sections: Active Queries, Recent Completions, Graph Traversal Stats,
Enrichment Queue
- Add query response time visualization (Chart.js or vanilla canvas)
- Add client-side filters for endpoint types and time ranges
- Include connection status indicator and manual refresh

Add dashboard serving:
- Add route `GET /dashboard` to metrics blueprint serving dashboard.html
- Configure Flask to serve static files from `automem/static/`
- Add CORS headers if needed
- Document dashboard access in README

Create terminal CLI:
- Create `scripts/metrics_cli.py` connecting to `/metrics/stream` with requests
stream=True
- Parse and format SSE events for terminal with color coding
- Display real-time log of queries, enrichment events, errors
- Add CLI arguments: --endpoint-type, --min-duration, --show-enrichment
- Implement connection retry with exponential backoff
- Document usage in `docs/METRICS_CLI.md`

Add documentation:
- Create `docs/OBSERVABILITY.md` with overview, architecture, usage examples
- Document SSE endpoint API contract (event types, schemas)
- Add environment variables: METRICS_BUFFER_SIZE (default 1000),
METRICS_ENABLE_PROFILING (default false)
- Document production deployment patterns
- Add troubleshooting section

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[jack-arturo]

Big chunk of this shipped back in January in commit 944168e: a real-time observability layer with a token-auth'd /stream SSE endpoint (automem/api/stream.py) that emits structured JSON events for memory.store, memory.recall, enrichment.start|complete|failed, and consolidation.run — each carrying elapsed_ms plus the relevant params/result counts. Plus scripts/automem_watch.py, a Rich TUI dashboard (run it with python scripts/automem_watch.py --url $URL --token $TOKEN) with panels for live events, consolidation timing, stats, errors, and a client-side garbage detector. Overhead stays near zero when nobody's subscribed.

So the Query Metrics section (incoming ops, params, response times) and most of the Enrichment section are covered. Not closing yet, though, because a non-optional part of the checklist isn't done:

• Graph Traversal Stats — nodes traversed, hops taken, relationship types followed, and server-side duplicate-detection counts. None of these are in the stream payloads. (The "duplicate" flag in the TUI is just a client-side md5 heuristic over the content preview, not real graph dedup counts.)
• Enrichment: queue depth and entity-extraction results aren't in the real-time stream (queue depth is only on the static /enrichment/status).
• Quality Metrics — the explicitly optional "if low-cost" tier — not implemented.

Leaving this open for whoever owns it to decide: if the streaming query/enrichment metrics are enough for your needs, feel free to close; otherwise the Graph Traversal Stats are the remaining ask, and would make a clean focused follow-up.