docs: proposed distributed Nebraska mode design doc

[Moustafa-Moustafa]

Adds docs/distributed-mode.md, the upstream proposed design doc for running Nebraska as one control node + N edge nodes with PostgreSQL logical replication of admin tables.

Refs: #1375

Proposed distributed Nebraska mode design doc

Adds docs/distributed-mode.md, the design doc consolidating the decisions from RFC #1375 for running Nebraska as one control node + N edge nodes with PostgreSQL logical replication of the admin tables. This is meant to be iterated on and updated based on the discussions and feedback.

How to use

Doce only, review the design and add any comments.,

Testing done

Not applicable.

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[Copilot]

Pull request overview

Adds a new design document describing a proposed “distributed Nebraska” deployment model (one control node plus multiple edge nodes) using PostgreSQL logical replication for admin metadata, while keeping runtime/telemetry data regional and writable.

Changes:

• Introduces docs/distributed-mode.md documenting the distributed topology, admin/runtime table split, and migration discipline for logical replication.
• Documents proposed code/package boundaries (pkg/api/admin, pkg/api/runtime, pkg/api/dbreads) and instance-mode behavior (403s for admin writes on non-control nodes).
• Defines an operator-driven replication/runbook and a proposed Postgres role/grants model.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ervcz]

Very nice write-up!

[ervcz]

@Moustafa-Moustafa
The document says replication is safe because foreign keys only point from runtime tables to admin tables. I think that is true for inserts and updates, but I am not sure about deletes: the apply worker runs with session_replication_role = replica, so triggers (including ON DELETE CASCADE) do not fire on the edge. So deleting an app or group on the control node could leave the edge’s local runtime rows (for example instance_application, activity, or the group_local sidecar) pointing at a row that no longer exists.

[ervcz]

One thing I did not see in the document: the in-memory caches. GetAppID and GetGroupID are only cleared by in-process add/update/delete, so on an edge (where admin changes arrive through replication) they may keep serving old data after a rename or delete on the control node. How are you thinking about cache invalidation on edges? What was the proposed solution to this - I don't reember? E.g. use Postgres LISTEN/NOTIFY from an ENABLE ALWAYS trigger, or just skip the cache in edge mode, or add a shorter TTL for the cache?