Background

VNC allows viewing and control of a host computer over a network. However, VNC's mouse support uses absolute mouse positions, and is ineffective when the host computer's mouse cursor is programmatically repositioned. This poses a challenge for use cases such as 3D video gaming, where relative mouse movements are used for camera control.

QEMU has an extension to the VNC protocol (RFB) for relative mouse movement, but this is not supported by other servers. RealVNC, too, has its own extension for relative pointer motion, but no free software implements this (see, e.g. TigerVNC issue #619).

I tried various combinations of client and server settings, Steam Remote Play, Synergy/Barrier and other protocols like SPICE, but none of these correctly handled relative pointer motion.

input-over-ssh

Note: This post refers to a legacy version of input-over-ssh. input-over-ssh has since been rewritten using a different framework.

To remedy this issue, I created input-over-ssh, a simple Python script to forward mouse and keyboard movement over SSH, using xinput and xdotool.

In forward.py, MOUSEHOLD_X and MOUSEHOLD_Y define the location at which the mouse cursor will be centered. Each frame, the client-side cursor will be reset to this position to measure relative movement. If MOUSEHOLD_W and MOUSEHOLD_H are non-zero, a Tkinter window of the given dimensions will be drawn around that location to prevent clicks interfering with other applications.

DEVICE_MOUSE and DEVICE_KEYBOARD give the names of the keyboard and mouse devices to monitor, as shown in the output of xinput list.

To run the script, pipe the output to SSH, ensuring that Python output is unbuffered using the -u flag:

python -u forward.py | ssh hostname.example.com bash

To exit, simply press Ctrl+Q.

Combining with VNC

This setup is easily combined with VNC in view-only mode to provide a full GUI experience.

On the host, disable the CursorPosUpdates/CursorShapeUpdates VNC extensions to force the server to directly render the cursor. For x11vnc, simply pass the -nocursorpos -nocursorshape flags.

On the client, enable view-only mode to prevent conflict between the VNC client and input-over-ssh. For TigerVNC vncviewer, simply pass the ViewOnly=1 flag.

Then run input-over-ssh as usual, and you should be able to control the remote cursor. This provides a lightweight, open-source alternative to services like Steam Remote Play.