SpectreMnemonic

SpectreMnemonic is an Elixir memory engine for live applications and agentic systems. It keeps hot working memory in ETS, links moments into a graph, persists durable memory through append-only stores, and recalls useful context with deterministic text matching, graph expansion, optional embeddings, and durable hybrid search.

It is not a replacement for your application database. It is the memory layer beside your framework: record what happened, recall nearby context, promote important moments, track stale or contradicted facts, and keep compact knowledge available without hydrating every old event.

{:ok, memory} =
  SpectreMnemonic.remember("Alice email is alice@example.com",
    stream: :chat,
    kind: :personal_fact,
    persist?: true
  )

{:ok, packet} = SpectreMnemonic.recall("Alice email")
{:ok, results} = SpectreMnemonic.search("Alice email")
{:ok, durable} = SpectreMnemonic.consolidate()

What It Gives You

• remember/2 for high-level intake: text, maps, parsed documents, chat, tasks, research notes, code notes, and tool events.
• signal/2 for low-level event recording when the caller already knows the stream, kind, task, and metadata.
• Active ETS memory for recent moments, task status, graph associations, artifacts, secrets, and action recipes.
• Deterministic local recall through keywords, entities, fingerprints, and graph expansion, even with no model configured.
• Optional embedding recall through an adapter or local Model2Vec provider.
• Built-in durable hybrid search over persisted records using BM25-style text scoring plus vector/signature reranking when embeddings exist.
• Scoped memory with optional scope, temporal validity fields, token-budget recall, and budgeted retrieval depth.
• Evidence-grounded observations and curated mental models for recall and reflection over memory.
• Governance state records: :candidate, :short_term, :promoted, :pinned, :stale, :contradicted, and :forgotten.
• Structured fact freshness and contradiction tracking for facts such as email, phone, age, status, birthday, deadline, and owner.
• Compact progressive knowledge in knowledge.smem.
• Encrypted secret memories with authorization-aware reveal.
• Plugs, adapters, and storage backends for framework-specific behavior.

Installation

Add the dependency:

def deps do
  [
    {:spectre_mnemonic, github: "elchemista/spectre_mnemonic"}
  ]
end

Start it as an OTP application or under your supervision tree. The default application starts ETS ownership, persistence, the durable index, stream routing, active focus, recall, consolidation, and the opt-in consolidation scheduler.

Quick Start

Use remember/2 for normal application memory:

{:ok, packet} =
  SpectreMnemonic.remember("TODO: implement durable graph search",
    title: "Planner note",
    stream: :planning,
    task_id: "alpha",
    scope: {:project, "alpha"},
    occurred_at: ~U[2026-05-30 10:00:00Z],
    metadata: %{source: :agent},
    persist?: true
  )

packet.root
packet.chunks
packet.summaries
packet.categories
packet.associations

Use recall/2 for active context:

{:ok, packet} =
  SpectreMnemonic.recall("how is alpha going?",
    scope: {:project, "alpha"},
    max_tokens: 2_000,
    budget: :mid
  )

packet.moments
packet.observations
packet.mental_models
packet.active_status
packet.associations
packet.knowledge

max_tokens is a best-effort packet budget. Recall may include one oversized primary evidence item when excluding it would make the packet empty.

Use search/2 when you want active recall plus durable persisted memory:

{:ok, results} = SpectreMnemonic.search("durable graph search", limit: 10)

Enum.map(results, &{&1.source, &1.family, &1.id, &1.score})

Use consolidate/1 to promote active memory into durable families:

{:ok, knowledge} = SpectreMnemonic.consolidate(min_attention: 1.0)

Use forget/2 to remove active memories and write tombstones:

SpectreMnemonic.forget({:task, "alpha"})
SpectreMnemonic.forget("mom_123")

Remember Plug Pipeline

remember/2 can run a composable plug pipeline before normal intake. This is the first extension point to reach for when SpectreMnemonic is embedded inside a larger framework.

Use plugs for framework-specific routing, classification, metadata, summarization, filtering, compression, secret detection, or replacing intake with a final custom packet.

Configure global plugs:

config :spectre_mnemonic,
  plugs: [
    MyApp.Memory.ProjectPlug,
    {MyApp.Memory.SecretRouterPlug, providers: [:github, :stripe]}
  ]

Add per-call plugs:

SpectreMnemonic.remember("sk_live_...",
  task_id: "chat-123",
  plugs: [MyApp.Memory.SessionPlug],
  secret_key: secret_key_32_bytes
)

Implement a plug:

defmodule MyApp.Memory.ProjectPlug do
  @behaviour SpectreMnemonic.Intake.Plug

  @impl true
  def call(memory, _opts) do
    %{
      memory
      | metadata: Map.put(memory.metadata, :project, :billing),
        tags: [:billing | memory.tags]
    }
  end
end

Plugs may continue, halt, or return a final packet, moment, secret, or signal. Low-level signal/2 does not run remember plugs.

Secret Memory

Secret memory is first-class because agents and live apps often see tokens, keys, passwords, credentials, or private notes while doing real work.

Secrets are stored as encrypted %SpectreMnemonic.Memory.Secret{} structs. The indexed text is redacted, and plaintext is encrypted before it enters active ETS or durable persistence.

Recommended flow:

1. A remember plug detects the secret.
2. The plug sets memory.secret? = true and memory.label.
3. SpectreMnemonic encrypts the original text.
4. Recall finds the redacted secret by label and metadata.
5. Reveal requires application authorization.

Low-level explicit secret storage:

{:ok, %{moment: secret}} =
  SpectreMnemonic.signal("github_pat_...",
    secret?: true,
    label: "GitHub token",
    secret_key: secret_key_32_bytes
  )

secret.text
#=> "secret: GitHub token"

Configure key access:

config :spectre_mnemonic,
  secret_key_fun: fn -> MyApp.Keys.memory_secret_key() end

Configure authorization:

config :spectre_mnemonic,
  secret_authorization_adapter: MyApp.SecretAuthorization

Reveal:

{:ok, revealed} =
  SpectreMnemonic.reveal(secret,
    secret_key: secret_key_32_bytes,
    authorization_adapter: MyApp.SecretAuthorization,
    authorization_context: %{user_id: current_user.id}
  )

If authorization is denied or missing, recall still succeeds and returns the locked redacted secret.

Core Concepts

Active Memory

Active memory is the hot working set in ETS. It stores signals, moments, associations, artifacts, action recipes, attention, and task status.

signal/2 writes one moment directly:

{:ok, %{moment: moment}} =
  SpectreMnemonic.signal("implemented disk replay checksum",
    stream: :task_execution,
    task_id: "alpha",
    kind: :task_execution,
    persist?: true,
    metadata: %{source: :agent}
  )

Intake Memory

remember/2 is the higher-level intake path. It normalizes input, creates a root moment, chunks long text, creates summaries and categories, extracts an entity timeline graph, and links the graph with typed associations.

The deterministic extractor handles names, ISO/month dates, simple events, emails, ages, numbers, and phone-like values. Phone-like values are redacted by default. Use:

SpectreMnemonic.remember(text, sensitive_numbers: :raw)
SpectreMnemonic.remember(text, sensitive_numbers: :skip)
SpectreMnemonic.remember(text, extract_entities?: false)

Memory can be scoped without changing the existing stream/task model. A scope is caller-owned data, such as a user, agent, tenant, or project tuple. Scoped recall only searches matching memory; unscoped recall stays broad for backward compatibility.

SpectreMnemonic.remember("Payment retry policy is stable",
  scope: {:tenant, "acme"},
  mission: :code_agent,
  extraction_mode: :concise,
  occurred_at: ~U[2026-05-01 12:00:00Z],
  valid_from: ~U[2026-05-01 00:00:00Z],
  persist?: true
)

SpectreMnemonic.recall("payment retry",
  scope: {:tenant, "acme"},
  valid_at: ~U[2026-05-30 00:00:00Z]
)

mission: is metadata by default. To let a mission affect intake retention, add the opt-in mission policy plug:

SpectreMnemonic.remember("TODO fix API retry contract",
  mission: :code_agent,
  plugs: [SpectreMnemonic.Intake.MissionPolicy]
)

The built-in :code_agent policy drops low-value conversational filler and prioritizes technical decisions, bugs, API contracts, constraints, TODOs, user preferences, and project state.

Temporal fields separate when something happened from when SpectreMnemonic learned it:

• :occurred_at - when the event or fact happened.
• :observed_at - when memory observed or learned it.
• :last_verified_at - when evidence was last verified.
• :valid_from and :valid_until - when a fact/model should be treated as true.

For richer extraction, configure an adapter:

config :spectre_mnemonic,
  entity_extraction_adapter: MyApp.MemoryExtractor

Adapters implement SpectreMnemonic.Intake.Extraction.Adapter and return graph fragments with entities, events, times, values, and relations.

Graph Associations

Memories can be linked manually:

SpectreMnemonic.link(source_id, :supported_by, target_id, weight: 0.8)

Recall expands through graph associations, so a task can bring in related research, code notes, artifacts, and action recipes.

Durable Persistence And Search

The default durable backend is an append-only local file store. Configure it explicitly when you want a custom data root:

config :spectre_mnemonic,
  persistent_memory: [
    write_mode: :all,
    read_mode: :smart,
    failure_mode: :best_effort,
    stores: [
      [
        id: :local_file,
        adapter: SpectreMnemonic.Persistence.Store.File,
        role: :primary,
        duplicate: true,
        opts: [data_root: "mnemonic_data"]
      ]
    ]
  ]

Persistent records are backend-neutral envelopes in families such as:

• :signals
• :moments
• :summaries
• :categories
• :embeddings
• :associations
• :knowledge
• :observations
• :mental_models
• :memory_states
• :consolidation_jobs
• :semantic_compaction_jobs
• :artifacts
• :action_recipes
• :tombstones

SpectreMnemonic.Persistence.Manager.replay/1 replays durable envelopes and applies tombstones.

Built-in Durable Hybrid Search

SpectreMnemonic keeps a rebuildable local durable index derived from replayed persistent records. The append-only store remains the source of truth.

The durable index scores with:

• BM25-style full-text scoring
• exact term overlap
• entity overlap
• vector cosine and binary-signature similarity when embeddings exist
• lifecycle boosts and demotions from :memory_states

Default visibility:

• :forgotten and :contradicted are hidden
• :stale is demoted
• :promoted is boosted
• :pinned is strongly boosted

The public entrypoint stays simple:

{:ok, results} = SpectreMnemonic.search("payment retry decision", limit: 10)

Rebuild the derived durable index if you manually changed durable storage:

SpectreMnemonic.Durable.Index.rebuild()

Observations, Mental Models, And Reflection

Observations are consolidated beliefs built from existing moments. Fact observations still come from governance facts, while deterministic V1 extraction also recognizes preferences, decisions, patterns, and project state. Observation type is stored in metadata as :observation_type so old %Observation{} records continue to work.

{:ok, observations} =
  SpectreMnemonic.consolidate_observations(scope: {:project, "alpha"})

{:ok, matches} =
  SpectreMnemonic.search_observations("payment retry",
    scope: {:project, "alpha"}
  )

{:ok, verified} =
  SpectreMnemonic.verify_observation(hd(observations),
    source_id: "mom_123",
    relation: :supports
  )

Mental models are curated stable answers for recurring queries. They are stored through the same persistence and durable search machinery as other memory.

{:ok, model} =
  SpectreMnemonic.put_mental_model(%{
    title: "Payment Retry Policy",
    query: "payment retry",
    answer: "Use bounded retries with idempotency keys.",
    scope: {:project, "alpha"},
    source_ids: ["mom_123"]
  })

{:ok, models} =
  SpectreMnemonic.search_mental_models("payment retry",
    scope: {:project, "alpha"}
  )

reflect/2 gathers mental models first, ranked observations second, then raw recall evidence. Observation evidence is ranked as decisions, preferences, project state, patterns, then facts. Without an adapter it returns a structured packet. With an adapter it normalizes the adapter output into packet.response. max_tokens is forwarded to recall as a best-effort packet budget and may include one oversized primary evidence item when excluding it would make the packet empty.

{:ok, packet} =
  SpectreMnemonic.reflect("What is the payment retry policy?",
    scope: {:project, "alpha"},
    max_tokens: 4_096
  )

packet.mental_models
packet.observations
packet.raw_memories
packet.citations
packet.response

Compaction

Physical compaction writes snapshots for append-only local files:

SpectreMnemonic.Persistence.Manager.compact(mode: :physical)

Semantic compaction asks a store or adapter to create compact records and tombstones:

SpectreMnemonic.Persistence.Manager.compact(mode: :semantic)
SpectreMnemonic.Persistence.Manager.compact(mode: :all)

Configure a semantic compaction adapter when your application wants custom, LLM-backed, or database-native compaction:

config :spectre_mnemonic,
  persistent_memory: [
    semantic_compact_adapter: MyApp.PersistentCompactAdapter,
    semantic_compact_families: [
      :moments,
      :knowledge,
      :summaries,
      :categories,
      :associations,
      :memory_states
    ],
    semantic_compact_limit: 1_000
  ]

Governance, Freshness, And Contradictions

Governance is stored as append-only :memory_states records so existing memory structs remain backward compatible.

Lifecycle states:

[:candidate, :short_term, :promoted, :pinned, :stale, :contradicted, :forgotten]

When a persisted moment is observed, SpectreMnemonic writes a lifecycle state. Consolidation promotes selected moments. Forgetting writes :forgotten.

Pin important memories:

SpectreMnemonic.signal("Payment retry policy is stable",
  persist?: true,
  memory_state: :pinned
)

Inspect state:

SpectreMnemonic.Governance.state_for("mom_123")

Structured Fact Upserts

SpectreMnemonic detects simple entity facts such as:

Alice email is alice@example.com
Deploy deadline is 2026-06-01
Task42 status is blocked

The upsert key is {normalized_subject, attribute}. A newer conflicting value marks the older fact :contradicted and promotes the newer fact.

{:ok, %{moment: old}} =
  SpectreMnemonic.signal("Alice email is old@example.com", persist?: true)

{:ok, %{moment: new}} =
  SpectreMnemonic.signal("Alice email is new@example.com", persist?: true)

SpectreMnemonic.Governance.state_for(old.id)
#=> :contradicted

SpectreMnemonic.Governance.state_for(new.id)
#=> :promoted

Pinned facts are not replaced automatically.

Provenance

Generated and persisted records carry provenance in metadata.provenance:

%{
  source_ids: ["mom_123"],
  source_span: nil,
  provider: :consolidator,
  confidence: 1.0,
  occurred_at: ~U[...],
  observed_at: ~U[...],
  last_verified_at: ~U[...],
  valid_from: ~U[...],
  valid_until: ~U[...]
}

Use provenance to explain why a recalled fact exists and whether it was generated, extracted, verified, or compacted.

Background Consolidation Scheduler

The scheduler is supervised but disabled by default. Enable it through config:

config :spectre_mnemonic,
  consolidation_scheduler: [
    enabled: true,
    interval_ms: 300_000,
    mode: :all,
    min_attention: 1.0,
    stale_after_ms: 30 * 24 * 60 * 60 * 1_000
  ]

Each tick can:

• run consolidation
• run freshness decay
• mark old unverified facts :stale
• compact persistent memory
• rebuild the durable search index

Check status:

SpectreMnemonic.ConsolidationScheduler.status()

Progressive Knowledge

knowledge.smem is a compact append-only knowledge log stored at data_root/knowledge/knowledge.smem. It is separate from active ETS memory and from the durable persistent-memory families.

Supported event types:

• :summary
• :skill
• :latest_ingestion
• :fact
• :procedure
• :compaction_marker

Append compact events:

SpectreMnemonic.Knowledge.Base.append(%{
  type: :skill,
  name: "Replay durable storage",
  text: "Use SpectreMnemonic.Persistence.Manager.replay/1 to inspect records.",
  metadata: %{attention: 2.0}
})

Teach a reusable skill:

{:ok, learned} =
  SpectreMnemonic.learn("""
  Debug local replay
  - inspect active.smem
  - check tombstones
  - compare replayed ids
  """)

learned.event.name
learned.event.steps

Search compact knowledge without loading the whole packet:

{:ok, matches} = SpectreMnemonic.search_knowledge("replay storage", limit: 5)

Load a budgeted packet:

{:ok, knowledge} =
  SpectreMnemonic.load_knowledge(
    max_loaded_bytes: 8_000,
    max_skills: 10,
    max_latest_ingestions: 10
  )

Compact progressive knowledge:

SpectreMnemonic.compact_knowledge()

Configure a custom compact adapter:

config :spectre_mnemonic,
  compact_adapter: MyApp.KnowledgeCompactAdapter

Embeddings

Embeddings are optional. Without an adapter, recall and search still work through text, fingerprints, graph associations, and durable BM25-style scoring.

Configure a custom adapter:

config :spectre_mnemonic,
  embedding_adapter: MyApp.EmbeddingAdapter

Adapters implement SpectreMnemonic.Embedding.Adapter.embed/2 and return {:ok, vector}, {:ok, embedding_map}, or {:error, reason}.

Enable the local Model2Vec provider:

config :spectre_mnemonic,
  embedding: [
    fast: [
      enabled: true,
      model_id: "minishlab/potion-base-8M",
      download: true
    ]
  ]

Downloads are opt-in. For production, pre-populate the cache or pass :model_dir.

Consolidation does not re-embed text. It copies the vector, binary_signature, and embedding already stored on each moment.

Action Recipes

Memories and artifacts can carry inert Action Language recipes. SpectreMnemonic stores and recalls these recipes as data only. It does not execute them.

{:ok, %{moment: moment, action_recipe: recipe}} =
  SpectreMnemonic.signal("cached weather JSON for Rome",
    action_recipe: "When Kinetic asks, refresh JSON from the weather endpoint",
    action_intent: "refresh cached JSON",
    ttl_ms: 60_000,
    refresh_on_recall?: true,
    source_url: "https://api.example.test/weather",
    tags: [:weather, :json]
  )

{:ok, packet} = SpectreMnemonic.recall("weather JSON Rome")
Enum.map(packet.action_recipes, & &1.text)

Execution is delegated only when you configure an adapter:

config :spectre_mnemonic,
  action_runtime_adapter: MyApp.KineticRuntime

Runnable Example

The example/ folder contains a local demo:

• parses test.txt, tasks.txt, and chat.txt
• writes a tiny local Model2Vec fixture
• remembers fixture events in parallel
• creates chunks, summaries, categories, extraction nodes, and graph edges
• persists active and consolidated records to append-only local storage
• demonstrates governed facts, contradiction, pinned/stale states, and durable hybrid search
• writes compact progressive knowledge
• registers an artifact
• runs replay, search, compaction, scheduler status, and evaluation output

Run it:

mix run example/demo.exs

Expected output includes lines like:

model        Smoke test vector_dims=4 signature_bytes=1
remembered   ... chunks=1 summaries=2 categories=... edges=...
governance   old=.../contradicted new=.../promoted pinned=.../pinned stale=.../stale
hybrid       source=persistent family=moments state=promoted score=...
knowledge    search "durable replay storage" -> ... compact matches
replay       Loaded ... records from .../example/mnemonic_data/segments/active.smem
compact      example_file snapshot=.../example/mnemonic_data/snapshots/snapshot-...
eval         size=6 recall_accuracy=... exact_fact_recall=... latency_ms=...

Generated runtime data goes under example/mnemonic_data/.

Evaluation And Development

Run the deterministic evaluation harness from IEx or your own test code:

SpectreMnemonic.Evaluation.run(size: 100)

It reports:

• recall accuracy
• exact fact recall
• latency in milliseconds

For development:

mix format
mix credo --strict
mix dialyzer
mix test

Project Layout

• lib/spectre_mnemonic.ex is the public facade.
• lib/spectre_mnemonic/active/* owns hot ETS focus, routing, and stream workers.
• lib/spectre_mnemonic/durable/* owns derived durable search indexes.
• lib/spectre_mnemonic/governance.ex owns lifecycle states, provenance, and structured fact contradiction logic.
• lib/spectre_mnemonic/observations.ex and lib/spectre_mnemonic/mental_models.ex own evidence-grounded observations and curated mental models.
• lib/spectre_mnemonic/reflection* builds reflection packets and delegates to optional reflection adapters.
• lib/spectre_mnemonic/consolidation_scheduler.ex owns opt-in background consolidation and freshness decay.
• lib/spectre_mnemonic/intake* powers remember/2, plugs, extraction, and intake packets.
• lib/spectre_mnemonic/recall/* builds recall packets, cues, fingerprints, and active embedding indexes.
• lib/spectre_mnemonic/knowledge/* loads knowledge.smem, compacts progressive knowledge, and consolidates active graph memory into durable families.
• lib/spectre_mnemonic/persistence/* coordinates durable stores, records, codecs, compaction, and storage behaviours.
• lib/spectre_mnemonic/embedding/* contains embedding adapters, vector math, binary quantization, and Model2Vec helpers.
• lib/spectre_mnemonic/actions/* delegates optional Action Language analysis and execution to an explicitly configured runtime adapter.