Video black on NVIDIA + niri: Portal ScreenCast negotiates LINEAR-only DmaBuf, `no more input formats` (wlr-direct input works)

[SpyrosPsarras]

Summary

On niri (Wayland) with an NVIDIA GPU, the wlr-direct strategy is selected correctly and input works perfectly (virtual keyboard + pointer), but video never renders - the RDP client shows a black screen. The PipeWire capture stream fails format negotiation with Error("no more input formats").

Root cause: for wlr-direct, video is acquired via standalone Portal ScreenCast and the stream is offered a DmaBuf format param with MOD_LINEAR only (MANDATORY|DONT_FIXATE). NVIDIA produces tiled/vendor-modifier buffers for screencast, so there is no modifier intersection, and the SHM fallback param is not accepted either.

I confirmed the display itself is capturable: grim (which uses wlr-screencopy) grabs the full screen fine on the same GPU. So this is specific to the Portal-ScreenCast + LINEAR-DmaBuf path, not the GPU or compositor.

Environment

• lamco-rdp-server 1.4.4-2 (Arch AUR, features pam-auth,h264,gui,wayland,libei,wl-clipboard)
• Compositor: niri 26.04 (Wayland, smithay-based)
• Portal backend running: xdg-desktop-portal-gnome (portal v5)
• GPU: NVIDIA GeForce RTX 2070 SUPER, driver 610.43.03
• Encoding: OpenH264 2.6.0 software (no hardware encode)
• Config: use_portals = false, [capture] protocol = "wlr", [input] input_protocol = "wlr", [performance] zero_copy = false, [hardware_encoding] enable_dmabuf_zerocopy = false

What works

• Strategy selection: Selected: wlr-direct strategy on niri
• Input: Bound zwp_virtual_keyboard_manager_v1 + Bound zwlr_virtual_pointer_manager_v1, real key/pointer events injected and observed on the host
• TLS (self-signed auto-generated), auth none, clipboard via wl-data-control, listening on 3389

The failure (server log)

INFO  server: wlr-direct: acquiring video via standalone Portal ScreenCast
INFO  server: Buffer type: DmaBuf (use_dmabuf=true)
INFO  pw_thread: Config: 2560x1440 @ 60fps, dmabuf=true, buffers=3
INFO  pw_thread: DmaBuf format param: 280 bytes (MOD_LINEAR, MANDATORY|DONT_FIXATE)
INFO  pw_thread: SHM fallback format param: 232 bytes
INFO  pw_thread: Format negotiation: 2 param(s) built (dmabuf=true)
INFO  pw_thread: Stream 136 state changed: Connecting -> Paused
INFO  pw_thread: Stream 136 state changed: Paused -> Error("no more input formats")
ERROR pw_thread: Stream 136 entered error state: no more input formats
WARN  display_handler: PipeWire stream error: no more input formats

Display pipeline then reports sent 0 (egfx: 0), dropped 0, skipped_damage 0 for the whole session - zero frames.

Proof the display is capturable (rules out GPU / compositor)

$ grim -l 0 /tmp/test.png    # grim uses wlr-screencopy
$ ls -l /tmp/test.png        # 41 MB, full 2560x1440 - success

Requests

1. For the wlr-direct strategy, allow video capture via wlr-screencopy instead of (or as a fallback to) Portal ScreenCast. The capability probe already reports WlrScreencopy: guaranteed - Direct capture without portal permission dialog on niri, but nothing routes video through it - use_portals=false and [capture] protocol=wlr have no effect on the standalone video path. This would fix the black screen and remove the per-start permission dialog.
2. Alternatively, offer the GPU's actual DmaBuf modifiers (not MOD_LINEAR only) so NVIDIA screencast can negotiate, or expose a config option to force SHM/MemFd capture buffers.

Secondary issue: restore token not persisted on niri

--grant-permission prints Permission granted and token stored!, but an immediately following --persistence-status reports Token Status: Not found (storage: KeePassXC). So every start re-prompts the ScreenCast permission dialog - unattended operation is not achievable on niri in this version.