fsgofer: add extension interface for custom backends#12950
Merged
Conversation
Contributor
Author
|
Just to get this on your radar @ayushr2 and @EtiennePerot . This depends on #12923, hence the added commit from there. The more relevant piece of proposal is this commit - f0bb46f. Happy to also start a design discussion in Github issues if that seems more fitting. No rush and appreciate your time. |
shayonj
force-pushed
the
s/gofer-backend-v2
branch
from
f0bb46f to
c56e256
Compare
shayonj
force-pushed
the
s/gofer-backend-v2
branch
4 times, most recently
from
6eed079 to
0d024ed
Compare
Contributor
Author
|
Just a gentle bump, in case there is any feedback for me? 😊 |
ayushr2
previously requested changes
May 7, 2026
Collaborator
|
BTW thanks a lot for upstreaming your work! This is amazing! |
copybara-service Bot
pushed a commit
that referenced
this pull request
May 14, 2026
The lisafs wire protocol already encodes the supported message list and the max message size per-mount on MountResp, but the server-side implementation reads both off a server-wide ServerImpl and reads ServerOpts off Server, so a custom implementation that needs to advertise different MIDs or run with different "OnDeleted" semantics has to bring its own lisafs.Server. That multi-Server shape carries its own renameMu, filesystem tree, and per-node opsMu instead of sharing them across mounts. This moves Mount, SupportedMessages, MaxMessageSize, and the former ServerOpts behind a per-Connection ConnectionImpl. Server keeps the shared renameMu, filesystem tree, connection lifecycle, and an opaque implementation value for existing fsgofer config access, while each Connection reads its supported MIDs from its ConnectionImpl and caches hot-path configuration such as max message size and ConnectionOpts at startup. Stock fsgofer behavior is unchanged because LisafsServer remains the server-wide config carrier and hands out a stock connection implementation. This is a precursor to a follow-up rework of #12950 that drops the multi-Server fan-out and lets LisafsServer choose a per-mount ConnectionImpl that returns the backend ControlFDImpl, all inside the single shared lisafs.Server. FUTURE_COPYBARA_INTEGRATE_REVIEW=#13158 from shayonj:s/lisafs-per-connection-opts ead4588 PiperOrigin-RevId: 915155303
copybara-service Bot
pushed a commit
that referenced
this pull request
May 14, 2026
The lisafs wire protocol already encodes the supported message list and the max message size per-mount on MountResp, but the server-side implementation reads both off a server-wide ServerImpl and reads ServerOpts off Server, so a custom implementation that needs to advertise different MIDs or run with different "OnDeleted" semantics has to bring its own lisafs.Server. That multi-Server shape carries its own renameMu, filesystem tree, and per-node opsMu instead of sharing them across mounts. This moves Mount, SupportedMessages, MaxMessageSize, and the former ServerOpts behind a per-Connection ConnectionImpl. Server keeps the shared renameMu, filesystem tree, connection lifecycle, and an opaque implementation value for existing fsgofer config access, while each Connection reads its supported MIDs from its ConnectionImpl and caches hot-path configuration such as max message size and ConnectionOpts at startup. Stock fsgofer behavior is unchanged because LisafsServer remains the server-wide config carrier and hands out a stock connection implementation. This is a precursor to a follow-up rework of #12950 that drops the multi-Server fan-out and lets LisafsServer choose a per-mount ConnectionImpl that returns the backend ControlFDImpl, all inside the single shared lisafs.Server. FUTURE_COPYBARA_INTEGRATE_REVIEW=#13158 from shayonj:s/lisafs-per-connection-opts ead4588 PiperOrigin-RevId: 915155303
copybara-service Bot
pushed a commit
that referenced
this pull request
May 14, 2026
The lisafs wire protocol already encodes the supported message list and the max message size per-mount on MountResp, but the server-side implementation reads both off a server-wide ServerImpl and reads ServerOpts off Server, so a custom implementation that needs to advertise different MIDs or run with different "OnDeleted" semantics has to bring its own lisafs.Server. That multi-Server shape carries its own renameMu, filesystem tree, and per-node opsMu instead of sharing them across mounts. This moves Mount, SupportedMessages, MaxMessageSize, and the former ServerOpts behind a per-Connection ConnectionImpl. Server keeps the shared renameMu, filesystem tree, connection lifecycle, and an opaque implementation value for existing fsgofer config access, while each Connection reads its supported MIDs from its ConnectionImpl and caches hot-path configuration such as max message size and ConnectionOpts at startup. Stock fsgofer behavior is unchanged because LisafsServer remains the server-wide config carrier and hands out a stock connection implementation. This is a precursor to a follow-up rework of #12950 that drops the multi-Server fan-out and lets LisafsServer choose a per-mount ConnectionImpl that returns the backend ControlFDImpl, all inside the single shared lisafs.Server. FUTURE_COPYBARA_INTEGRATE_REVIEW=#13158 from shayonj:s/lisafs-per-connection-opts ead4588 PiperOrigin-RevId: 915155303
copybara-service Bot
pushed a commit
that referenced
this pull request
May 16, 2026
Earlier, the lisafs package allowed programming varying server implementations. However, it assumed that all connections on the same server have the same implementation. In an effort to build custom filesystem gofer implementations, it would be useful to run various implementations on the same server instance. The server synchronizes concurrent RPCs on the same filesystem tree by maintaining the entire filesystem tree in memory and holding server-side locks. We would want to have these concurrency guarantees across different implementations. This change largely refactors all the implementation specificity from Server to Connection. The lisafs wire protocol already encodes the supported message list and the max message size per-mount on MountResp. So no wire protocol changes needed. Hence this is completely backwards compatible and should not lead to any change in behavior. This is a precursor to a follow-up rework of #12950 that drops the multi-Server fan-out and lets LisafsServer choose a per-mount ConnectionImpl that returns the backend ControlFDImpl, all inside the single shared lisafs.Server. Co-authored-by: Shayon Mukherjee <shayonj@gmail.com> PiperOrigin-RevId: 915234536
copybara-service Bot
pushed a commit
that referenced
this pull request
May 16, 2026
Earlier, the lisafs package allowed programming varying server implementations. However, it assumed that all connections on the same server have the same implementation. In an effort to build custom filesystem gofer implementations, it would be useful to run various implementations on the same server instance. The server synchronizes concurrent RPCs on the same filesystem tree by maintaining the entire filesystem tree in memory and holding server-side locks. We would want to have these concurrency guarantees across different implementations. This change largely refactors all the implementation specificity from Server to Connection. The lisafs wire protocol already encodes the supported message list and the max message size per-mount on MountResp. So no wire protocol changes needed. Hence this is completely backwards compatible and should not lead to any change in behavior. This is a precursor to a follow-up rework of #12950 that drops the multi-Server fan-out and lets LisafsServer choose a per-mount ConnectionImpl that returns the backend ControlFDImpl, all inside the single shared lisafs.Server. Co-authored-by: Shayon Mukherjee <shayonj@gmail.com> PiperOrigin-RevId: 916313093
shayonj
force-pushed
the
s/gofer-backend-v2
branch
4 times, most recently
from
e4d4b50 to
28a89ba
Compare
shayonj
changed the title
shayonj
force-pushed
the
s/gofer-backend-v2
branch
4 times, most recently
from
47ef47e to
b694d54
Compare
shayonj
force-pushed
the
s/gofer-backend-v2
branch
from
b694d54 to
1ca361e
Compare
Building a custom gofer (e.g. for network-backed storage, encrypted filesystems, or tiered caches) currently requires forking the runsc binary and copying/maintaining unexported setup and seccomp code. This adds an Extension interface that lets custom filesystem backends register with the stock gofer and serve LisaFS connections for specific mounts without forking. The interface follows the socket.Provider-style registration pattern: NewConnection returns a nil lisafs.ConnectionImpl to decline a mount, and the first registered extension that returns a non-nil implementation handles it. NewConnection receives the sandbox's OCI runtime spec, the specific mount being served, the resolved mount path, and readonly state, so extensions can read sandbox-wide configuration from spec.Annotations and per-mount configuration from the mount itself without a side-channel. Stock fsgofer remains the default when no extension claims a mount. All mounts still share one lisafs.Server; extensions only choose the per-connection lisafs.ConnectionImpl and lisafs.ConnectionOpts now supported by lisafs.Server.CreateConnection. SeccompRules lets extensions declare additional syscalls, merged with the stock allowlist before installation. Zero behavior change when no extensions are registered: the stock fsgofer path runs unchanged, identical to today. This follows the same pattern as the network plugin: inactive when not configured, no impact on the default path. New package runsc/fsgofer/extension defines the Extension interface and registration. The gofer command iterates registered extensions for each mount before falling through to fsgofer. The seccomp filter install path accepts extra rules for merging extension rules with the stock allowlist. Also adds documentation in g3doc/user_guide/filesystem.md and pkg/lisafs/README.md describing how to use the extension interface.
shayonj
force-pushed
the
s/gofer-backend-v2
branch
from
1ca361e to
7139233
Compare
ayushr2
approved these changes
May 20, 2026
copybara-service Bot
pushed a commit
that referenced
this pull request
May 20, 2026
Building a custom gofer (e.g. for network-backed storage, encrypted filesystems, or tiered caches) currently requires forking the runsc binary and copying/maintaining unexported setup and seccomp code. This adds an `Extension` interface that lets custom filesystem backends register with the stock gofer and serve LISAFS connections for specific mounts without forking. This builds on #13180, which moved LISAFS implementation selection and connection options to the connection. With that in place, custom backends can plug into the stock gofer without creating separate `lisafs.Server` instances. All mounts continue to share one `lisafs.Server`, preserving the server-side filesystem tree and synchronization across stock and extension-backed mounts. Registered extensions are queried in order for each mount. `NewConnection` returns a nil `lisafs.ConnectionImpl` to decline a mount, and the first extension that returns a non-nil implementation handles it. `NewConnection` receives the sandbox's OCI runtime spec, the specific `*specs.Mount` being served, the resolved mount path, and readonly state, so extensions can read sandbox-wide configuration from `spec.Annotations` and per-mount configuration from the mount itself without a side channel. Stock `fsgofer` remains the default when no extension claims a mount. Extensions only choose the per-connection `lisafs.ConnectionImpl` and `lisafs.ConnectionOpts` now supported by `lisafs.Server.CreateConnection`. `SeccompRules` lets extensions declare additional syscalls, merged with the stock gofer allowlist before installation. There are no behavior changes when no extensions are registered. Also adds documentation in `g3doc/user_guide/filesystem.md` and `pkg/lisafs/README.md` describing how to use the extension interface. FUTURE_COPYBARA_INTEGRATE_REVIEW=#12950 from shayonj:s/gofer-backend-v2 7139233 PiperOrigin-RevId: 918225708
copybara-service Bot
pushed a commit
that referenced
this pull request
May 20, 2026
Building a custom gofer (e.g. for network-backed storage, encrypted filesystems, or tiered caches) currently requires forking the runsc binary and copying/maintaining unexported setup and seccomp code. This adds an `Extension` interface that lets custom filesystem backends register with the stock gofer and serve LISAFS connections for specific mounts without forking. This builds on #13180, which moved LISAFS implementation selection and connection options to the connection. With that in place, custom backends can plug into the stock gofer without creating separate `lisafs.Server` instances. All mounts continue to share one `lisafs.Server`, preserving the server-side filesystem tree and synchronization across stock and extension-backed mounts. Registered extensions are queried in order for each mount. `NewConnection` returns a nil `lisafs.ConnectionImpl` to decline a mount, and the first extension that returns a non-nil implementation handles it. `NewConnection` receives the sandbox's OCI runtime spec, the specific `*specs.Mount` being served, the resolved mount path, and readonly state, so extensions can read sandbox-wide configuration from `spec.Annotations` and per-mount configuration from the mount itself without a side channel. Stock `fsgofer` remains the default when no extension claims a mount. Extensions only choose the per-connection `lisafs.ConnectionImpl` and `lisafs.ConnectionOpts` now supported by `lisafs.Server.CreateConnection`. `SeccompRules` lets extensions declare additional syscalls, merged with the stock gofer allowlist before installation. There are no behavior changes when no extensions are registered. Also adds documentation in `g3doc/user_guide/filesystem.md` and `pkg/lisafs/README.md` describing how to use the extension interface. FUTURE_COPYBARA_INTEGRATE_REVIEW=#12950 from shayonj:s/gofer-backend-v2 7139233 PiperOrigin-RevId: 918225708
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Building a custom gofer (e.g. for network-backed storage, encrypted filesystems, or tiered caches) currently requires forking the runsc binary and copying/maintaining unexported setup and seccomp code. This adds an
Extensioninterface that lets custom filesystem backends register with the stock gofer and serve LISAFS connections for specific mounts without forking.This builds on #13180, which moved LISAFS implementation selection and connection options to the connection. With that in place, custom backends can plug into the stock gofer without creating separate
lisafs.Serverinstances. All mounts continue to share onelisafs.Server, preserving the server-side filesystem tree and synchronization across stock and extension-backed mounts.Registered extensions are queried in order for each mount.
NewConnectionreturns a nillisafs.ConnectionImplto decline a mount, and the first extension that returns a non-nil implementation handles it.NewConnectionreceives the sandbox's OCI runtime spec, the specific*specs.Mountbeing served, the resolved mount path, and readonly state, so extensions can read sandbox-wide configuration fromspec.Annotationsand per-mount configuration from the mount itself without a side channel.Stock
fsgoferremains the default when no extension claims a mount. Extensions only choose the per-connectionlisafs.ConnectionImplandlisafs.ConnectionOptsnow supported bylisafs.Server.CreateConnection.SeccompRuleslets extensions declare additional syscalls, merged with the stock gofer allowlist before installation.There are no behavior changes when no extensions are registered.
Also adds documentation in
g3doc/user_guide/filesystem.mdandpkg/lisafs/README.mddescribing how to use the extension interface.