fix(core): Multi-display, Wayland parity, Computer Use, and recording

[ehfd]

Coordinated changes across selkies (server + web clients + dashboard) and pixelflux (Rust capture/encode). pcmflux is unchanged. The through-line is closing X11↔Wayland and WebSockets↔WebRTC parity gaps while adding an independent recording pipeline and an HTTP Computer-Use control surface.

Multi-display

• X11 logical monitors are now swapped atomically under XGrabServer (delete-then-recreate), so RandR 1.5 never exposes a partial monitor set to the WM and secondary displays maximize correctly. Fixes the stale-geometry maximize bug and the dead cursor-press.
• Wayland multi-output capture reaches parity with X11 (per-output damage + cursor overlay).
• Single-secondary remains a client-architecture fact on both transports — the WS and WR cores each wire only #display2; #display3+ connects as primary. A second #display2 cleanly supersedes the first (WS: data-server takeover; WR: signaling eviction 4001 — same outcome).

Computer Use

• New HTTP control endpoint. X11 uses XTEST; Wayland uses evdev button codes behind a shared backend seam.
• Off-layout typing: characters absent from the active keymap are synthesized via a temporary overlay mapping (X11) / virtual-keyboard keymap slots (Wayland), collision-safe under concurrent input and byte-exact.

Recording

• Independent, crash-truncation-safe fragmented-MP4 pipeline in pixelflux (wall-clock 90 kHz tfdt), controllable via env var, the Computer-Use channel, and the API. Verified GPU zero-copy tap on the Wayland path (system libgbm).

Clipboard, input, IME

• First-use clipboard now works on both backends; the Wayland fix sets data-device + primary-selection focus so native paste is no longer dead.
• Image clipboard round-trips on both WS and WR (full parity).
• Input fixes: Ctrl+Key IME, AltGr over XTEST, stuck-Ctrl, non-English input, and HiDPI scaling. Keyboard-layout hint is precise on Chromium and language-derived elsewhere (new keyboard-layout.js).

Stream lifecycle / viewer pause

• Consumer-aware STOP_VIDEO fixes the shared-tab freeze — capture only stops when all consumers of a display are paused.
• Per-peer WR viewer pause on tab-hide gates that peer's sender only; capture stops solely when every consumer is paused, and resumes with an IDR. Verified this does not regress the controller or the primary client (a hidden viewer leaves the controller advancing).

Dashboard

• Fullscreen-toggle visibility control, shortened citation, and translation strings.

Compatibility & testing

• Python 3.9–3.14+, CUDA/NVENC 11–13+. NVENC concurrent-session cap is respected with graceful CPU/x264 fallback past the limit (cap is a GPU property, so identical for WS and WR by construction).

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[thelamer]

I don't like this from a code maintenance standpoint, the recording sink was introduced out of an external PR here #15. Since its introduction it is something I have never used and I doubt many of our users are. It requires a session to be running and producing frames to even function with the most reliable way to use it by flagging the stream with streaming mode so it is always producing frames. The introduction of the keyframes is to support recording it as a raw .h264 stream and not re-encoding with ffmpeg.

So down the line for v2.0.0+ I do not want it to be another thing that is needed to be regression tested against hevc, av1, etc encoders. We should just mem copy, get some normal keyframes like a recording would expect, and spit out exactly what the browser client is getting from a frame standpoint.

In general the whole system needs a makeover to be something useful, the initial idea is we are getting "free" recording ability leveraging existing tech, but to be truly useful it kind of needs to be it's own pipe. A user that wants to record a session really just wants an mp4 to be spit out in the end and it really should not depend on active clients. This makes it useful for CI scenarios and any other agentic automation, actions are performed in the desktop session and you start and stop the recording while those actions are being performed, then you get a properly encoded video file that can be used out of the box to present the data in a web browser or anywhere else.

As this tech stack exists it is really only useful impo for corporate envs where you are recording your users to audit them.

[ehfd]

@thelamer Will you take it if I reimplement it on a maintainable way?

[thelamer]

I mean do you share my concern? Like do we really need this to be embedded in all the encoders ?

Lets just intercept the the delivery pipeline for now and ask for keyframes when needed. The final implementation of this I see it being able to run independently of selkies completely thus the env var PIXELFLUX_. just in general it would be nice if all setting can auto adapt to using env vars, pythonic stuff , or cli flags like the selkies settings.

Eventually land on where you can make a very short python script to spin up a wayland socket, use the rest api to control it via CU, and record what happens in the session. A more general purpose selenium or pyautogui type thing than a full desktop delivery stack to a web browser. Use this for a battle tested virtual display manager with gpu support to run desktop applications in test envs for x11 and wayland. I think the accepted solution for many of these is just to run xvfb right now.

But I don't want to develop this all now, lets get 2.0.0 cut, lets get you the wheels you need to build out the next iteration of Selkies, and lets earmark this for complete refactor while it still provides the same basic functionality even if degraded it used to.

[ehfd]

I mean do you share my concern? Like do we really need this to be embedded in all the encoders ?

Of course not. I agree.

Let me propose a compromise, then.