Skip to content

Commit d57ea14

Browse files
Paul Kyleclaude
andcommitted
feat: watcher fault isolation (ADR-002), nightly consolidation, migration --review, prompt versioning
Watcher hardening: - Event handlers catch exceptions — one bad file never stalls the indexer - vec0 writes use delete-then-insert (virtual tables don't support OR REPLACE) - ADR-002 documents the design decision and extensibility pattern Nightly consolidation: - run_nightly() — lightweight daily pass, UPDATE/SUPERSEDE only - NightlyConfig in config with lookback_days and allowed_ops - palinode consolidate --nightly (CLI + API) - Dedicated nightly-consolidation.md prompt OpenClaw migration: - Scoring-based type detection (heading weighted 3x over body) - --review flag for interactive type confirmation before writing - review_callback for programmatic use Prompt versioning: - palinode prompt list/show/activate CLI commands - API endpoints for prompt management - Frontmatter-based versioning with active flag Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent d4acf82 commit d57ea14

20 files changed

Lines changed: 1670 additions & 32 deletions

ADR-001-tools-over-pipeline.md

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# ADR-001: Tools Over Pipeline
2+
3+
**Status:** Accepted
4+
**Date:** 2026-04-01
5+
**Context:** Claude Mythos announcement, Nate B Jones "compensating complexity" framework
6+
7+
## Decision
8+
9+
Palinode's primary value is its **17 MCP tools + file-based storage + git versioning**, not its 4-phase injection pipeline. The pipeline is scaffolding for current models. The tools survive any model upgrade.
10+
11+
## Context
12+
13+
Nate B Jones' framework distinguishes:
14+
- **Outcome specs** — what you want (survives model upgrades)
15+
- **Procedures** — how to do it (breaks on model upgrades)
16+
- **Tools** — capabilities the model can use (survives)
17+
- **Compensating complexity** — workarounds for model limitations (disposable)
18+
19+
Applied to Palinode:
20+
21+
### What survives any model
22+
- `palinode_search` — find relevant memories
23+
- `palinode_save` — store typed memories
24+
- `palinode_session_end` — capture session outcomes
25+
- `palinode_list/read` — browse the memory directory
26+
- `palinode_diff/blame/rollback/push` — git operations
27+
- `palinode_history/timeline` — file-level provenance
28+
- `palinode_trigger` — prospective recall
29+
- `palinode_entities` — entity graph
30+
- `palinode_consolidate` — trigger compaction
31+
- `palinode_lint` — structural health checks
32+
- `palinode_ingest` — capture from URLs
33+
- `palinode_status` — system health
34+
- Deterministic executor (validates LLM output, applies ops)
35+
- Markdown files as source of truth
36+
- Git versioning
37+
- Security scanning (business rule)
38+
39+
### What's compensating complexity (today's scaffolding)
40+
- 4-phase auto-injection (Phase 1-4) — compensates for models that don't know to search
41+
- Trivial message skip list — compensates for models that can't judge query relevance
42+
- Layer split keyword heuristics — compensates for models that can't classify by content
43+
- `json_repair` — compensates for models that output malformed JSON
44+
- Explicit JSON format instructions in compaction prompt — same
45+
46+
## Consequences
47+
48+
### Positioning
49+
Lead with: "17 tools + a memory directory your agent uses however it wants."
50+
Not: "4-phase injection pipeline."
51+
52+
The pipeline is an implementation detail. The tools are the interface.
53+
54+
### Architecture evolution
55+
1. **v0.5 (now):** Pipeline + tools. Pipeline does automatic injection. Tools available for explicit use.
56+
2. **v1.0:** Pipeline becomes optional/configurable. Default to tools-first for capable models. Pipeline as fallback.
57+
3. **v2.0:** Tools-only mode. Model decides what to retrieve, when to consolidate. Pipeline removed or opt-in legacy.
58+
59+
### What stays regardless of version
60+
- MCP tool interface (the API contract)
61+
- File-based storage (markdown + git)
62+
- Deterministic executor (LLM proposes, executor disposes)
63+
- Security guardrails
64+
- `core: true` flagging (user intent, not model workaround)
65+
66+
### What to audit on each model upgrade
67+
Run the compaction pipeline with prompt sections deleted. Measure what gets worse vs what stays the same. Remove what the new model makes unnecessary.
68+
69+
## References
70+
- Nate B Jones, "Every workaround you built for the last model is now breaking the next one" (April 2026)
71+
- Internal design discussion on tools-first architecture
72+
- Cursor's Planner-Worker-Judge convergence (same structural pattern across Anthropic, DeepMind, OpenAI)

ADR-002-watcher-fault-isolation.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# ADR-002: Fault Isolation in the Indexing Layer
2+
3+
**Status:** Accepted
4+
**Date:** 2026-04-06
5+
**Deciders:** Paul Kyle
6+
7+
## Decision
8+
9+
Palinode's indexing layer treats every file event as an independent unit of work. A failure to index one file must never prevent indexing of subsequent files. Database write operations against SQLite virtual tables (`vec0`, `fts5`) use explicit delete-then-insert rather than relying on conflict-resolution clauses.
10+
11+
## Motivation
12+
13+
Palinode's value proposition depends on a simple invariant: **if a file exists in `PALINODE_DIR`, it is searchable.** The file watcher daemon is the component responsible for maintaining this invariant. It runs unattended, often as a systemd service, processing file events indefinitely.
14+
15+
Daemons that process streams of independent events have a well-known fragility: if one event raises an unhandled exception, the processing thread dies while the main process stays alive. The system appears healthy — the PID exists, the port responds, the service status is green — but no new work is being done. This is the worst class of failure because it is **silent and persistent**.
16+
17+
The indexing layer must therefore be designed around two principles:
18+
19+
1. **Event isolation** — each file event is processed in its own error boundary
20+
2. **Defensive writes** — database operations use patterns that work reliably across all SQLite table types, including virtual tables with non-standard semantics
21+
22+
## Design
23+
24+
### Event isolation
25+
26+
Every watchdog event handler wraps its processing in a catch-all that logs and continues:
27+
28+
```
29+
on_modified / on_created → try _process_file() except log + continue
30+
on_deleted → try delete_chunks() except log + continue
31+
```
32+
33+
A failed file is skipped. It will be retried automatically on the next modification, or manually via `palinode ingest`. The invariant is temporarily broken for that one file but holds for everything else.
34+
35+
### Defensive virtual table writes
36+
37+
SQLite virtual tables implement their own storage engines via `xUpdate`. The SQLite documentation notes that virtual tables "may or may not" support conflict resolution in INSERT statements. In practice:
38+
39+
- **`vec0`** (sqlite-vec) does not reliably handle `INSERT OR REPLACE` — it can raise a UNIQUE constraint error on an existing primary key instead of replacing the row
40+
- **`fts5`** has similar limitations with external-content tables
41+
42+
The safe pattern for any virtual table upsert:
43+
44+
```sql
45+
DELETE FROM virtual_table WHERE id = ?; -- no-op if absent
46+
INSERT INTO virtual_table (id, ...) VALUES (?, ...);
47+
```
48+
49+
This is two statements instead of one. The cost is negligible — embedding generation dominates the write path by orders of magnitude.
50+
51+
### Extensibility
52+
53+
These patterns apply to any new index type added to Palinode. If a future version adds a graph index, a keyword index, or a secondary vector store, the same rules hold:
54+
55+
- Wrap writes in the event handler's error boundary (already provided by the catch-all)
56+
- Use delete-then-insert for any virtual table or extension-managed storage
57+
- Use `INSERT OR REPLACE` only for regular SQLite tables where it is guaranteed
58+
59+
No per-index error handling is needed in calling code — the event boundary catches everything.
60+
61+
## Trade-offs
62+
63+
| | Benefit | Cost |
64+
|---|---|---|
65+
| **Event isolation** | One bad file never stalls the daemon | A persistently failing file logs on every modification until root-caused |
66+
| **Delete-then-insert** | Works on all table types, past and future | One extra statement per write (negligible) |
67+
| **Catch-all in handlers** | Maximum availability | Broad exception handling can mask programming errors in development |
68+
69+
The catch-all is acceptable because the watcher is a **production daemon**, not application logic. Availability takes priority over fail-fast. Errors are always logged with the file path and exception, so they remain debuggable.
70+
71+
## Consequences
72+
73+
- The indexing layer is resilient by default — new indexes inherit fault isolation without additional code
74+
- Watcher logs contain structured error lines for any failed file, enabling monitoring and alerting
75+
- The `chunks`, `chunks_fts`, and `chunks_vec` write paths are now consistent in their error-handling strategy
76+
- No schema changes or migrations required
77+
78+
## References
79+
80+
- [SQLite Virtual Table Interface — xUpdate](https://www.sqlite.org/vtab.html#xupdate)
81+
- [sqlite-vec](https://github.com/asg017/sqlite-vec) — the `vec0` virtual table extension
82+
- [watchdog](https://github.com/gorakhargosh/watchdog) — filesystem event library

examples/CLAUDE.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,7 @@ This project uses Palinode for persistent memory (MCP server: palinode).
3030
- Raw code (git handles that)
3131
- Step-by-step debug logs (save the resolution, not the journey)
3232
- Trivial changes ("fixed typo" — not worth a memory)
33+
34+
### If MCP is not connected:
35+
- Use CLI equivalents: `palinode search "query"`, `palinode save "content" --type Decision`
36+
- Check connection: `palinode status` or `palinode_status()`

0 commit comments

Comments
 (0)