Skip to content

paxos: applied_barriers ledger is never pruned and grows unbounded across reconfiguration #629

Description

@SebastianThiebaud

The paxos state machine's applied_barriers ledger is never pruned. It is a HashMap<u64, u64> (state_machine.rs:66, mirrored in HighWaterSnapshot.applied_barriers at log_entry.rs:61) keyed by node id; the doc claims its size is "bounded by cluster size," which holds only for a static cluster. OmniPaxos supports reconfiguration, and across reconfigurations the set of node ids that ever appended a Barrier grows monotonically with no eviction site (confirmed: no removal anywhere). The same unbounded map is serialized into every snapshot and shipped in snapshot transfer. Today the blast radius is nil — the standalone bootstrap fixes configuration_id: 1 and paxos membership is Unsupported — but it becomes a slow unbounded growth of snapshot payload and apply-fold cost the moment reconfiguration is enabled. Cheap to fix now while the surface is small.

Scope

  • crates/tsoracle-driver-paxos/src/state_machine.rs:66 and crates/tsoracle-driver-paxos/src/log_entry.rs:61 — prune ledger entries for node ids absent from current membership at snapshot-creation time (or cap to the active voter set), so the map cannot accrete entries for departed nodes.

Exit criteria

  • cargo test -p tsoracle-driver-paxos passes; the linearization/barrier-seq invariants (max_logged_barrier_seq recovery) are preserved.
  • A test that simulates membership churn asserts the ledger does not retain entries for removed node ids across a snapshot.

Explicit non-goal

Implementing paxos dynamic membership itself — this issue only ensures the barrier ledger is bounded by current membership so the leak is pre-empted before reconfig lands.

Parent:

Metadata

Metadata

Assignees

No one assigned

    Labels

    reliabilityImprove stability, fault tolerance, and predictable behavior in real-world/runtime conditionstech debtCleanup or hardening work that addresses shortcuts or deferred engineering quality issues

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions