Liftbridge 26.01.1

Liftbridge v26.01.1

Breaking Changes

• Versioning: Changed from semantic versioning (v1.x.x) to CalVer (YY.MM.PATCH)
• Go version: Minimum requirement changed from 1.21 to 1.25.6
• Version variable: server.Version changed from const to var for build-time injection

New Features

Reverse Subscription Support (#290)

Added the ability to subscribe to streams in reverse order (newest to oldest messages).

Use Cases:

• Consumer cursor optimization: Finding the latest cursor position is now O(k) instead of O(n), where k is the number of updates per cursor vs n total messages
• Recent events display: UIs can show the most recent messages first without scanning the entire stream
• Debugging: Quickly examine recent messages when troubleshooting issues
• Event search: Find recent occurrences of specific events by reading backwards

API Changes:

• Added Reverse boolean field to SubscribeRequest protobuf message
• When Reverse=true, messages are delivered from newest to oldest
• Works with all start positions (LATEST, EARLIEST, OFFSET, TIMESTAMP)
• Compatible with both committed and uncommitted reads

Implementation:

• Added ReverseReader and MessageReader interface in commitlog package
• Added reverseIndexScanner for backward index traversal
• Added reverseSegmentScanner for backward segment reading
• Optimized getLatestCursorOffset() to use reverse subscription internally

Example (using go-liftbridge):

// Subscribe in reverse order starting from the latest message
client.Subscribe(ctx, "my-stream", func(msg *lift.Message, err error) {
    fmt.Println(msg.Offset(), string(msg.Value()))
}, lift.StartAtLatestReceived(), lift.Reverse())

GitHub Actions Release Workflow

Added automated release workflow triggered by release/* branches:

• Multi-platform binaries: darwin/amd64, darwin/arm64, linux/amd64, linux/arm64, windows/amd64, windows/arm64
• Linux packages: Debian (.deb) and RPM (.rpm) for amd64/arm64
• Docker images: Multi-arch images pushed to ghcr.io/liftbridge-io/liftbridge
• Automated testing: Runs full test suite, binary verification, and Docker image tests before release
• Draft releases: Creates GitHub draft release with all artifacts and checksums

Package Installation

Linux packages now include:

• Systemd service file (/lib/systemd/system/liftbridge.service)
• Default configuration (/etc/liftbridge/liftbridge.yaml)
• Data directory (/var/lib/liftbridge/data)
• Dedicated liftbridge user/group

Anonymous Telemetry

Added opt-out telemetry to help improve Liftbridge by collecting anonymous usage statistics.

What's Collected:

• Instance ID (random UUID, persistent per installation)
• Liftbridge version
• OS information (name, version, architecture)
• CPU cores (physical/logical)
• Total system memory

Privacy:

• No message data, stream names, or personally identifiable information
• No network addresses, credentials, or performance metrics
• Data sent to telemetry.basekick.net every 24 hours
• Easy opt-out via configuration or environment variable

Configuration:

telemetry:
  enabled: false  # Set to false to disable
  interval:
    seconds: 86400  # Default: 24 hours

Or via environment variables:

export LIFTBRIDGE_TELEMETRY_ENABLED=false

See the Telemetry Documentation for full details.

Performance Optimizations & Benchmark Tools

Added benchmark tools and server-side optimizations achieving 241K msgs/sec throughput.

Benchmark Results (single node, RF=1, 256-byte messages):

Configuration	Throughput	Latency (P99)
1 publisher, synchronous	~30K msgs/sec	306µs
1 publisher, pub-batch=100	~139K msgs/sec	1.0ms
4 publishers, pub-batch=100	~241K msgs/sec	2.0ms
256 publishers, synchronous	~200K msgs/sec	1.4ms

For comparison, NATS JetStream achieves ~220K msgs/sec with similar settings.

Optimizations:

• Fast path for RF=1 partitions (skips commit queue for LEADER/NONE ack policies)
• Timer-based message batching support

Benchmark Tools:

• bench/producer - Producer benchmark with HDR histogram latency tracking
• bench/consumer - Consumer benchmark for throughput measurement
• Support for concurrent publishers and async batching (--pub-batch flag)

Dependencies Updated

Package	Before	After
github.com/hashicorp/raft	v1.1.2	v1.7.3
github.com/hashicorp/go-hclog	v0.14.1	v1.6.2
github.com/liftbridge-io/nats-on-a-log	v0.0.0-20200818	v0.0.0-20251217

Code Changes

• Fixed raft bootstrap with RaftMaxQuorumSize: ensure local server is always
  a voter to support raft v1.7.3's pre-vote protocol requirement
• Migrated CI from CircleCI to GitHub Actions
• Updated server/version.go to support build-time version injection via ldflags

Docker Updates

• Updated all Dockerfiles to Go 1.25-alpine
• Updated dev-cluster NATS image from 2.6.4 to 2.12.3
• Updated dev-cluster base image from debian:stretch-slim to debian:bookworm-slim
• Added VERSION build argument for proper version tagging
• Optimized Dockerfile layer caching
• Removed obsolete docker/circleci/ folder

Bug Fixes

Segment Deletion Edge Case

Fixed a bug where partial deletion failure could leave the system in an inconsistent state.

Problem: If segment file deletion failed partway through (e.g., due to permissions or I/O error), some segments would be deleted while others remained, but an error was returned. This could cause data inconsistency.

Solution: Implemented a mark-then-delete approach:

1. All segments are first marked as deleted (atomic per-segment), removing them from the read path
2. File deletion is then attempted for each segment
3. If file deletion fails, segments are already marked deleted and won't be visible to readers
4. Remaining files will be cleaned up on the next cleanup cycle

Changes:

• Added deleted flag to segment struct with MarkDeleted() and IsDeleted() methods
• Added deleteSegments() helper to deleteCleaner implementing the two-phase approach
• Refactored applyMessagesLimit, applyBytesLimit, and applyAgeLimit to use the new helper

Corrupt Index File Recovery (#411)

Fixed a startup panic when index files become corrupted.

Problem: If an index file became corrupted (e.g., due to unclean shutdown, disk errors), Liftbridge would panic on startup with "corrupt index file" error and fail to start. This left users unable to recover without manual intervention.

Solution: Implemented automatic index rebuild from the log file:

1. When corruption is detected during index initialization, the corrupt index is deleted
2. A new index is created and rebuilt by scanning the log file
3. All valid message entries are re-indexed from the log data
4. Server startup proceeds normally with the rebuilt index

Changes:

• Added rebuildIndex() method to segment that scans log file and recreates index entries
• Modified setupIndex() to catch errIndexCorrupt and attempt automatic recovery
• Added tests for corrupt index detection and recovery scenarios

Snapshot Restore Panic (#414)

Fixed a startup panic when restoring from a Raft snapshot.

Problem: When a node started and restored state from a Raft snapshot, it would panic with a nil pointer dereference at partition.go:1311. This occurred because partitions were trying to start leader/follower loops before the API server was initialized.

Solution: Mark streams and consumer groups as "recovered" during snapshot restore, deferring their startup until finishedRecovery() is called after log replay completes. This ensures s.api is initialized before partitions attempt to start.

Changes:

• Modified Restore() in fsm.go to pass recovered=true to applyCreateStream() and applyCreateConsumerGroup()

Signal Handling Race with Embedded NATS (#373)

Fixed a race condition when using embedded NATS that could prevent graceful shutdown.

Problem: When Liftbridge runs with embedded NATS (EmbeddedNATS: true), both servers registered their own signal handlers for SIGINT/SIGTERM. Whichever handler ran first would win - if NATS won, it would call os.Exit() immediately, preventing Liftbridge from performing graceful shutdown (closing partitions, draining NATS connections, stopping Raft).

Solution: Set opts.NoSigs = true when creating the embedded NATS server, disabling NATS's signal handling and allowing Liftbridge to handle all signals for proper graceful shutdown.

Changes:

• Modified startEmbeddedNATS() in server.go to set opts.NoSigs = true

Leader Not In ISR Panic (#354)

Fixed a nil pointer dereference when a partition leader is not in the ISR.

Problem: When restoring from a Raft snapshot, if the snapshot contained inconsistent state where a partition's leader was not in its ISR (In-Sync Replicas), the server would panic with a nil pointer dereference at partition.go:812. This could happen if a node was removed from the ISR via ShrinkISR but a snapshot was taken before a new leader election occurred.

Solution: Added a defensive nil check in becomeLeader(). If the leader is not found in the ISR, it logs a warning and adds itself to the ISR with the current offset, allowing the server to recover from...

v1.9.0

Changelog

Added

• Support ARM64 Linux by @tylertreat in #394
• Support consumer group events in activity stream by @tylertreat in #398
• Support ACL authorization on stream level (beta) by @LaPetiteSouris in #388

Changed

• Change default timeout for internal broker info requests to 3 seconds by @tylertreat in #401
• Remove override logic for tls-client-* flags since these flags were never exposed by @tylertreat in #408

Fixed

• Fix potential for serving stale cursors by @tylertreat in #395
• Fix link in consumer groups documentation by @mihaitodor in #397
• Fix data race on consumer assignments by @tylertreat in #400
• Fix flaky tests and race conditions by @tylertreat in #403 and #404
• Fix panic caused by partition queue dispose by @tylertreat in #405

Full Changelog: v1.8.0...v1.9.0

v1.8.0

Changelog

Added

• Consumer Groups by @tylertreat in #387
• Add cursors.stream.replication.factor configuration by @tylertreat in #387

Changed

• Allow cursors.stream.auto.pause.time to be changed by @tylertreat in #391
• Prevent clients from creating or deleting reserved streams by @tylertreat in #391

Full Changelog: v1.7.1...v1.8.0

v1.7.1

Changelog

Fixed

• Fix partition leader failover by @tylertreat in #383
• Support MacOS ARM64 architecture by @tylertreat in #378

Full Changelog: v1.7.0...v1.7.1

v1.7.0

Changelog

Added

• Expose information about server's loads on FetchMetadata by @LaPetiteSouris in #362

Changed

• Update to go 1.17 and update nats-server dependency by @tylertreat in #368
• Update docusaurus for website by @tylertreat in #370
• Update Dockerfile go versions by @tylertreat in #374

Fixed

• Fix FetchCursor blocking by @tylertreat in #366
• Fix bug related to stream deletion by @tylertreat in #367
• Fix bug related to commit log high watermark file by @tylertreat in #367
• Fix race with raftLogListeners mutex by @tylertreat in #369

Full Changelog: v1.6.0...v1.7.0

v1.6.0

Changelog

Added

• Add a Raft log listener, contributed by @Jmgr (#336)
• Add support for encrypting data at rest on server side, contributed by @LaPetiteSouris (#338)

Changed

• Validate NATS subject in CreateStream (#345)

Fixed

• Fix possibility for HW to move backwards (#337)
• Fix flaky replication test, contributed by @LaPetiteSouris (#341)
• Fix StartAtTime EOF error (#349)
• Fix memory leak in replication loop, contributed by @dvolodin7 (#351)

v1.5.1

Changelog

Fixed

• Fix deadlock issue when there is a leader change for a partition but there is no metadata leader (#329)
• Fix replication notify flood leading to high CPU usage on idle partitions (#329)

v1.5.0

Changelog

Added

• Add clustering.replication.max.bytes config (#308)
• Add support for running embedded NATS server (#309)
• Add support for limiting Raft quorum size (#312)
• Add support for optimistic concurrency control, contributed by @LaPetiteSouris (#296)

Changed

• Update standalone Docker image to use embedded NATS server (#310)
• Change partition replicas and leader selection to distribute assignment based on load (#324, #326)

Fixed

• Fix race with Raft node initialization (#325)

v1.4.1

Changelog

Fixed

• Fix FetchPartitionMetadata debug log (#301)
• Unmap memory mapped indexes on close (#303)

v1.4.0

Changelog

Added

• Add read-only status and event timestamps to metadata, contributed by @Jmgr (#276)
• Implement subscription stop position, contributed by @alexrudd (#282, #286)

Changed

• Remove unused Dockerfile directives, contributed by @alexrudd (#281, #288)
• Change dev-standalone host default to 0.0.0.0 (#287)

Fixed

• Correct FetchPartitionMetadata debug log, contributed by @alexrudd (#278)
• Drain acks when client closes PublishAsync RPC (#284)
• Fix loopback client race (#285)
• Fix delete stream error when dir is not empty (#299)