Durable consumers Acknowledgment Floor "reset" on power failure

[MirkoGibin]

Observed behavior

I'm not sure whether this is a defect or expected behavior. I'm opening this issue to collect more information, as I haven't been able to find any related details in the documentation.

Our setup is a single-node deployment with no replication enabled.

The NATS server is receiving and processing events. At some point, the physical server is shut down (for any reason) and then started again.

After the restart, we observe that durable consumers associated with streams using the Limits retention policy have their acknowledgement floor reset to 1. This causes a full replay of all messages in the corresponding streams.

We noticed that this behavior only affects consumers related to streams that, during server startup, log messages like the following:

[WRN] Filestore [streamName] Stream state outdated, last block has additional entries, will rebuild
[WRN] Filestore [streamName] Recovering stream state from index errored: prior state file ...

Is this an expected behavior when a stream state rebuild happens, or could this indicate an issue with the stream/consumer state recovery?
Could it be related to sync_interval configuration?

Expected behavior

Durable consumer should be able to maintain AckFloor after restore

Server and client version

nats server v2.12.5

Host environment

• Nats Jetstream v2.12.5 on a single node installation on a physical server
• Kubernetes deployed as RKE2
• File based storage backed by a Longhorn PVC
• OS: Linux RHEL compatible

Steps to reproduce

No response

[MirkoGibin]

Other details I collected in the meanwhile:

• If I set sync_interval to 10s or 2m and I see some consumer having AckFloor reset to 1
• If i set sync_interval to always I see come consumer having a last sequence ahead of the stream last sequence (like reported here Consumer State Reconciliation After Stream Data Loss (R1 Consumers) #7693 or here Consumer Sequence ahead of Stream Sequence #6600)
• It looks like only Limits policy stream are affected by these behaviors.

[aku-ato]

same to me on version 2.12.1

[ilCollez]

Hi, wanted to share a related incident we experienced on the same day (2026-04-15) that involves the same ack floor desync mechanism — and notably, also triggered by a power outage.

Our scenario

We run a 3-node NATS cluster (v2.12.1, file storage on Kubernetes with OpenEBS PVCs, WorkQueue retention). During the night of April 14–15 we had a significant power outage. Servers came back online at 08:39 UTC, and our JOB_STREAM sequences restarted from 1 at that exact moment — consistent with NATS rebuilding the filestore state after the ungraceful shutdown. However, the 7 durable pull consumers were left untouched, their ack floor frozen at the old high-water mark (~3.4M stream sequence).

The result was the mirror image of what you're describing: instead of the ack floor resetting to 1, ours stayed far above the new stream sequences (1–5,961). NATS considered every new message already processed (Unprocessed: 0 on all consumers), so no worker ever received a delivery. The symptom was completely silent: no errors, no redeliveries, just workers polling forever and getting nothing.

Last Delivered Message: Consumer sequence: 654,189  Stream sequence: 3,408,244
Acknowledgment Floor:   Consumer sequence: 654,189  Stream sequence: 3,408,244
Unprocessed Messages: 0   ← stream had 5,961 messages, all "invisible"

Key differences from your issue

	Your case	Our case
Trigger	Power failure + server restart	Power failure + server restart
Ack floor direction	Resets down to 1 → full replay	Stays above new stream sequences → silent starvation
Retention policy	Limits	WorkQueue
Cluster	Single-node	3-node cluster
NATS version	v2.12.5	v2.12.1
Observable symptom	Duplicate/replayed messages	Zero message delivery

Resolution

We manually deleted all 7 consumers and restarted the worker pods, which forced them to recreate the consumers with a fresh ack floor. Delivery resumed immediately.

The two cases seem to be two sides of the same coin: after an ungraceful restart, the stream state and the consumer state can diverge in either direction, with no built-in detection. In our case there was no warning in the NATS server logs (we didn't observe a "will rebuild" message — though we haven't been able to fully rule it out yet). The only diagnostic path was manually inspecting nats consumer info and comparing the ack floor against the stream's first sequence.

Hope this adds useful context to the investigation.

[olmaqurrent]

Hello, adding a third data point to this issue.

• 3-node cluster, Limits retention, MaxMsgsPerSubject: 1 (used as last-value cache for per-recipient state distribution, ~600+ filtered consumers).
• NATS v2.12.0
• Trigger was an Azure VM live-migration causing a ~13-second pause on one node, not a full power loss. Cluster maintained quorum throughout.
• Symptom: the consumer never delivers anything, but its tracked position in the stream is already past the message it's supposed to deliver. The message is clearly there — nats stream view shows it sitting at the consumer's filter subject — but because the consumer thinks it has already moved past it, the server reports num_pending=0 and the client just waits forever. delivered.consumer_seq=0 (nothing was ever handed out), delivered.stream_seq points beyond the matching message, num_waiting>0. Same shape as @ilCollez case (consumer position ahead of where matching messages live → silent starvation), just at much smaller sequence numbers and starting from a never-delivered state rather than after a long run of deliveries.
• We tested two relevant DeliverPolicy values (DeliverLast, DeliverLastPerSubject) — bug fires on each. So it's not specific to the per-subject metadata lookup we initially suspected.
• Mitigated operationally via per-client reconnect-driven consumer recreation triggered on every JWT-cycle reconnect (~hourly).

Since we have MaxMsgsPerSubject: 1 we also changed DeliverPolicy to DeliverAll, not semantically correct for our use case but it works and is more robust.

[SerAlxVeeam]

Hi, wanted to share a related incident we experienced on the same day (2026-04-15) that involves the same ack floor desync mechanism — and notably, also triggered by a power outage.

Funny. We’re seeing the same situation with a single node.

[scalalang2]

I experienced this issue in this week. I think this is very critical