#518 lever 4 (durable-load tail) -- yielding snapshot. Parent: #518 / #507.
Problem (verified): save_shard_local (crates/ironcache/src/coordinator.rs:1101-1114) holds the shard's store borrow across the ENTIRE dump_shard_keyspace loop + fsync, so SAVE and BGSAVE BLOCK the dumping shard for its whole keyspace dump + fsync (documented at crates/ironcache-persist/src/lib.rs:17-21). A 100k-key shard at 512-key chunks = ~196 chunks ~= up to ~1s block -> a p99.9 spike whenever a save aligns with writes. Redis fork-stalls, Dragonfly snapshot-spikes; the moat is a BOUNDED, PREDICTABLE save tail.
Fix: yield the shard between dump chunks. snapshot_chunk already RELEASES the store borrow between chunks (persist.rs:93-94), so the caller CAN yield: acquire the borrow per-chunk (not across the whole loop), tokio::task::yield_now().await between chunks so the drain loop services queued writes; make the ICSAVE branch of run_remote (+ run_local_save) async.
DESIGN SUBTLETY (must decide + document): yielding + interleaving writes means the snapshot is NO LONGER a clean point-in-time (some keys pre-write, some post-write mid-dump). Decide + document the consistency semantics: for a CACHE this is likely acceptable (an approximate restore point), but it MUST be a conscious, documented decision, not an accident -- and the crash-safety invariant (manifest written LAST, so a torn dump never loads) MUST hold. If strict point-in-time is required, this needs versioning/COW instead (bigger; defer). State which you chose and why.
Acceptance: a save yields between chunks so writes complete during BGSAVE (test: queue writes during a save, assert they are serviced, not blocked to the end); crash-safety preserved (manifest-last kill test: truncate a dump, assert boot refuses/ignores it per #530); p99.9 tail under a write+BGSAVE mixed workload drops measurably; the snapshot-consistency decision documented. No dashes.
#518 lever 4 (durable-load tail) -- yielding snapshot. Parent: #518 / #507.
Problem (verified):
save_shard_local(crates/ironcache/src/coordinator.rs:1101-1114) holds the shard's store borrow across the ENTIREdump_shard_keyspaceloop + fsync, so SAVE and BGSAVE BLOCK the dumping shard for its whole keyspace dump + fsync (documented at crates/ironcache-persist/src/lib.rs:17-21). A 100k-key shard at 512-key chunks = ~196 chunks ~= up to ~1s block -> a p99.9 spike whenever a save aligns with writes. Redis fork-stalls, Dragonfly snapshot-spikes; the moat is a BOUNDED, PREDICTABLE save tail.Fix: yield the shard between dump chunks.
snapshot_chunkalready RELEASES the store borrow between chunks (persist.rs:93-94), so the caller CAN yield: acquire the borrow per-chunk (not across the whole loop),tokio::task::yield_now().awaitbetween chunks so the drain loop services queued writes; make the ICSAVE branch of run_remote (+ run_local_save) async.DESIGN SUBTLETY (must decide + document): yielding + interleaving writes means the snapshot is NO LONGER a clean point-in-time (some keys pre-write, some post-write mid-dump). Decide + document the consistency semantics: for a CACHE this is likely acceptable (an approximate restore point), but it MUST be a conscious, documented decision, not an accident -- and the crash-safety invariant (manifest written LAST, so a torn dump never loads) MUST hold. If strict point-in-time is required, this needs versioning/COW instead (bigger; defer). State which you chose and why.
Acceptance: a save yields between chunks so writes complete during BGSAVE (test: queue writes during a save, assert they are serviced, not blocked to the end); crash-safety preserved (manifest-last kill test: truncate a dump, assert boot refuses/ignores it per #530); p99.9 tail under a write+BGSAVE mixed workload drops measurably; the snapshot-consistency decision documented. No dashes.