Reclaim metastore BoltDB disk space without requiring restart #5279

Problem

Metastore index cleanup can delete old index buckets and add tombstones, but the underlying BoltDB file does not shrink in a long-running process. Disk space is only reclaimed when a snapshot is restored and metastore.snapshot-compact-on-restore=true.

In practice, this means long-running single-binary/self-hosted deployments can keep a large metastore.boltdb file even after retention cleanup has removed old data from the logical index.

This is especially problematic for local filesystem deployments with finite disk, where the process may run until disk exhaustion and crashloop before compaction ever happens.

Code pointers

• pkg/metastore/fsm/boltdb.go
  • bbolt.Compact is only called from the snapshot restore path.
  • compaction is gated by SnapshotCompactOnRestore.
• pkg/metastore/fsm/fsm.go
  • metastore.snapshot-compact-on-restore defaults to false.
  • snapshots are persisted from the running BoltDB state, but regular online compaction is not performed.
• pkg/metastore/index_raft_handler.go
  • TruncateIndex deletes logical index shards, but this does not shrink the BoltDB file.

Proposal

Add a mechanism to reclaim metastore BoltDB disk space for long-running metastores without requiring a restart/snapshot restore cycle.

Possible approaches:

• Periodically compact the BoltDB database in the background when safe.
• Compact after successful snapshots outside of the restore path.
• Provide an explicit admin/API operation to trigger metastore DB compaction.
• Track BoltDB file size vs logical/live size and compact only above a threshold.

Acceptance criteria

• A long-running metastore can reclaim BoltDB disk space after retention cleanup without process restart.
• The mechanism does not block raft apply/read paths for excessive time.
• The mechanism is safe for single-node and multi-node raft deployments.
• Metrics/logging expose when compaction runs, how long it takes, and compaction ratio.
• Add tests for compaction behavior or at least the triggering/gating logic.