Persist zones by ximon18 · Pull Request #550 · NLnetLabs/cascade

ximon18 · 2026-03-26T00:42:16Z

Status

Zone edits (e.g. via file edit and zone reload or due to changes received via XFR) are persisted both for loaded and signed zones. On application restart persisted zones are restored and cascade zone status and dig AXFR appear to work as they did prior to application shutdown.

The code has been initially reviewed and led to some wanted changes:

@bal-e will extend the zonedata crate interface to enable moving some of the changes to Cascade itself.
Invoke save_now() instead of mark_dirty(). This is because persisting zone data to disk not enough, as if there was a power outage between calling mark_dirty() and publication of the zone the paths to the persisted files (that are recorded in state) would not have been written to disk, thus save_now() should be invoked instead. We may actually want to invoke save_now() in the publication server itself to ensure all published zone related state is persisted, but that is out of scope for this PR. Update: It's not clear to me that this is actually needed.

tl;dr

Extend src/persistence/persist.rs to write snapshots and deltas to disk (currently in DNS AXFR/IXFR wire format).
Extend src/persistence/restore.rs to restore zones by loading the zone snapshot into a zone replacer and passing the zone diffs one at a time to a zone patcher.
Extend src/persistence/ to store generated/restored diffs for use by IXFR out.
Extend ZoneState with two vecs of paths to persisted snapshot and delta files.
Extend the CLI to show the zone is restoring in cascade zone status output.

Known issues:

No support (yet) for condensing/compacting/purging old persisted deltas.
Binary snapshot/delta format is not inspectable by operators. Could add a `cascade debug subcommand to render a snapshot/delta in XFR presentation format.
There's no "scheduling" or spreading out of zone restoration activity, like refreshing it happens as soon as possible.
There are no metrics for the restoration process (yet).
Snapshots and deltas are loaded entirely into memory, while they could be processed in a streaming manner instead.
The persistence location is hard-coded.

Alternative snapshot/delta storage formats

A choice would need to be driven by requirements:

Do we need to support IXFR out at the loaded review nameserver? If so, is just one delta enough? Actually do we even need to keep a loaded diff, a review hook would only need it for the next received zone update, not after restart.
What do we tune for? Memory usage? Persistence speed? Restoration speed?

Some ideas:

Presentation XFR format: was not done in this PR due to the pain of having to convert new base to old base to render in presentation format, and parsing DNS records in presentation format is only currently properly supported by domain/cascade for entire zones, though one could use the "treat a single RR line as a zonefile" hack that the Stelline code uses.
(De)serialize cascade DiffData to JSON/CBOR/whatever. Would tie the format to the internal format, changing the internal format would require writing additional code to handle storage format migration or support parsing old as well as new format. A PRO of this approach is being able to storage additional meta data with the snapshot/delta, if needed.

If you are changing Rust code or integration tests (Cargo.*, crates/, etc/, integration-tests/, src/):
- Did you run the integration tests with act through the act-wrapper (as described in TESTING.md)?

bal-e

@ximon18 and I had a discussion while I was writing this review, and decided that I will add commits changing the PR to implement a new approach for the zone data storage state machine. I've left these comments here for us to come back once I'm done; I also didn't review everything, so there will be more to come.

bal-e · 2026-04-07T08:41:01Z

+
+    /// The index of the current loaded instance.
+    pub(super) curr_loaded_index: bool,
+
+    /// The index of the current signed instance.
+    pub(super) curr_signed_index: bool,


Are these fields necessary, if this state is only used at the very beginning (where they would both be 0/false)?

bal-e · 2026-04-07T08:41:37Z

    pub(super) curr_signed_index: bool,
 }

+pub struct UninitializedStorage {


This type (and RestoringPersistedStorage below) need documentation, similar to the other types in this file. I'm happy to write it by adding to the PR.

bal-e · 2026-04-07T08:42:11Z

        assert!(
            old_reviewer.loaded_index == self.curr_loaded_index,
-            "'old_reviewer' does not point to the current instance",
+            "'old_reviewer' does not point to the current instance: {} vs:{}",


{} vs:{} -> {} vs {}?

bal-e · 2026-04-07T08:46:10Z

+        let viewer = unsafe { ZoneViewer::new(self.data.clone(), true, true) };
+
+        let signed = unsafe { &*self.data.signed[0].get() };
+        let serial = signed.soa.as_ref().unwrap().rdata.serial;


I don't think the serial number should be extracted here. Perhaps we need methods on LoadedZoneBuilt and SignedZoneBuilt that give you readers so you can observe the data, or just getters for the serial?

bal-e · 2026-04-07T08:48:28Z

+        assert!(
+            Arc::ptr_eq(&signed_built.data, &self.data),
+            "'built' is for a different zone"
+        );


You should also check loaded_built. And the assert message should clarify which parameter it is referring to.

bal-e · 2026-04-07T09:51:42Z

+        let (transition, state) = if let ZoneDataStorage::Uninitialized(s) = state {
+            let s = s.initialize();
+            t.move_to(ZoneDataStorage::Passive(s));
+            transition(&mut self.state.storage.machine)
+        } else {
+            (t, state)
+        };


I'd prefer the above commented-out initialize() function over this silent automatic conversion. Especially since it should only occur at startup and we can handle it outside this function.

With the current setup, we might actually want to do this check in other places, but it is hard to tell.

bal-e · 2026-04-07T09:54:00Z

+                let (s, loaded_reviewer, signed_reviewer, viewer, serial) =
+                    s.finish(loaded_built, signed_built);
+                self.state.storage.loaded_reviewer = loaded_reviewer;
+                self.state.storage.signed_reviewer = signed_reviewer;


This code does not correctly handle the reviewer types. They should be passed back into the storage state machine, so it can guarantee that viewers only exist for the permitted 0/1 instances. We should discuss what the right adjustments to the storage state machine's flow are.

bal-e · 2026-04-07T09:58:47Z

+    /// The zone is in an empty initial uninitialized state pending possible
+    /// restoration from persisted state or initialization directly to an
+    /// empty passive state.
+    Uninitialized(UninitializedStorage),


It might be better to omit this state entirely and only have a Restoring state, which can then decide to proceed with or skip restoring. Restoring might already need that functionality, e.g. if the previously persisted instances could not be found.

bal-e · 2026-04-07T09:59:19Z

+    Uninitialized(UninitializedStorage),
+
+    /// The zone is being restored from persisted state.
+    Restoring(RestoringPersistedStorage),


The enum variant name should match the underlying name, for consistency. I think the underlying type should be RestoringStorage.

bal-e · 2026-04-07T09:59:56Z

@@ -86,9 +94,6 @@ pub enum ZoneDataStorage {
 impl ZoneDataStorage {
    /// Construct a new [`ZoneDataStorage`].
    pub fn new() -> (Self, LoadedZoneReviewer, SignedZoneReviewer, ZoneViewer) {


Perhaps we want to change this function so it no longer returns reviewers? That might be easier all around.

…al system test.

Will need changing based on PR #589.

Currently fails due to an unrelated TSIG bug causing dig to fail.

- Merges in the ixfr-out and full-signer-update-state branches as these were needed for local testing, but should be obsoleted / synced with main when PRs #631 ("Save last serial and key tags in zone state")and #605 ("Add IXFR out support") get merged. - Extends the persist-restore system test to cover more cases. Should perhaps be split out into separate smaller tests. - Actually adds restored diffs to the zone storage to be served by IXFR out. Will need updating to match the changes/fixes that have since been made in PR #605. - Fixes an issue where persisted diffs from multiple SOA serials, e.g. 1..2..3 would be condensed on restore so that only a single IXFR diff from 1..3 would be available instead of two diffs from 1..2 and 2..3 being available. - Clears the set of known persisted data file paths for a zone if any of those files are missing or cannot be parsed during restoration.

As step.if doesn't seem to be honoured by nektos/act.

Not via TCP but via single SOA response, per RFC 1995. Also adds a system test using dig.

…allocation.

The signed diff is not available when the loaded diff is restored but should be stored with the corresponding loaded diff, so make the signed diff an Option to be made Some as soon as it is restored.

- Don't hold the zone state lock while restoring as it blocks the zone status command. - Log restoring started at INFO level as well as restoring complete. - Only log restoring complete if restoring actually happened. - Log the zone being restored at INFO level, not just TRACE level.

Otherwise zone status of a restored zone doesn't show that it is published.

…een extracted to PR #650.

Validated with the ixfr-out system test from PR #650.

ximon18 added this to the 0.1.0-beta1 milestone Mar 26, 2026

ximon18 added the enhancement New feature or request label Mar 26, 2026

ximon18 requested review from Philip-NLnetLabs, bal-e and tertsdiepraam March 27, 2026 09:12

ximon18 linked an issue Mar 30, 2026 that may be closed by this pull request

Zone persistence #295

Open

bal-e reviewed Apr 7, 2026

View reviewed changes

bal-e mentioned this pull request Apr 7, 2026

Integrate the zone server(s) with the new zone storage #552

Merged

1 task

ximon18 mentioned this pull request Apr 10, 2026

Cascade doesn't save state on SIGTERM. #573

Closed

bal-e mentioned this pull request Apr 15, 2026

Implement control flow for restoring zones #589

Merged

1 task

ximon18 added 7 commits April 28, 2026 15:48

Add limited initial IXFR out support (AXFR fallback only) and a minim…

2c1f606

…al system test.

PoC IXFR-out.

5dd79aa

Will need changing based on PR #589.

Merge branch 'main' into ixfr-out

fd6c2f3

Clippy.

9421203

De-clutter ixfr-out system test output.

5345a2b

Show TSIG key of source in output of CLI command zone status.

f8890f7

Updated to match updated persist/restore framework.

52bd50d

ximon18 force-pushed the persist-zones branch from a80510c to 52bd50d Compare May 1, 2026 06:57

ximon18 and others added 12 commits May 6, 2026 10:28

Clippy.

c613e24

Handle errors during restore of persisted zone data.

c4b91d5

Merge branch 'main' into persist-zones

0222445

Save last serial and key tags in zone state.

39b0b20

Improvements to the persist-restore system test.

b18906e

Don't persist if nothing changed.

6b8548b

Account for a current known issue until it is resolved.

fda163e

Switch to apex_remove and apex_extra.

5c3c6cf

Fmt.

ce23145

More work on the persist-restore system test.

e655ea1

Merge branch 'main' into persist-zones

cccdf75

Merge branch 'full-signer-update-state' into persist-zones

0887919

ximon18 added 28 commits May 8, 2026 13:36

(WIP) Extend the ixfr-out test to actually verify produced output.

cd989d1

Currently fails due to an unrelated TSIG bug causing dig to fail.

Add more logging of request handling w.r.t. ACL processing.

c733215

FIX: include loaded RRs in the IXFR out, not just signed RRs.

4e48270

Merge branch 'main' into ixfr-out

f0f9ebb

Adjust test condition.

d399f1d

As step.if doesn't seem to be honoured by nektos/act.

Review feedback: signal upgrade to TCP on UDP IXFR request.

cfd91be

Not via TCP but via single SOA response, per RFC 1995. Also adds a system test using dig.

Review feedback: Duplicate spawn comment.

01fac00

Review feedback: Note the potential to use flat_map() to avoid a vec …

7905990

…allocation.

Review feedback: Remove irrelevant comment.

02efdfc

Verify more XFR outputs against expectations.

baf4c64

Comment out code.

e730fad

Merge branch 'main' into persist-zones

825afd3

Restore both loaded and signed IXFR diff parts.

2c52bcf

The signed diff is not available when the loaded diff is restored but should be stored with the corresponding loaded diff, so make the signed diff an Option to be made Some as soon as it is restored.

Forget & remove persisted zone data if it cannot be loaded.

a7888aa

FIX: Persist zone state last_published metadata.

45abfe9

Otherwise zone status of a restored zone doesn't show that it is published.

Clippy.

2fc7fac

Rename diffs -> diff_paths.

492efa1

Rename old_request -> request.

db02b5f

Merge branch 'main' into persist-zones

d442640

Remove early merged versions of the ixfr-out test, these have since b…

765d26f

…een extracted to PR #650.

Merge fix.

df39500

Validated with the ixfr-out system test from PR #650.

Remove change to system-tests that isn't needed by this PR.

da67628

Add back accidentally removed comment.

fabd541

Remove unnnecessary temporary diagnostic Debug derive.

97ed92b

Remove duplicate code resulting from merge.

4f9d86d

Remove reference to ixfr-out system test as that now has its own PR.

f9f4bfa

ximon18 marked this pull request as ready for review May 12, 2026 14:53

ximon18 mentioned this pull request May 13, 2026

Prune persisted zone data #655

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Persist zones#550

Persist zones#550
ximon18 wants to merge 56 commits into
mainfrom
persist-zones

ximon18 commented Mar 26, 2026 •

edited

Loading

Uh oh!

bal-e left a comment

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

bal-e Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ximon18 commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status

tl;dr

Known issues:

Alternative snapshot/delta storage formats

Uh oh!

bal-e left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ximon18 commented Mar 26, 2026 •

edited

Loading