🌒

Proxmox + Nvidia + LXC + Docker + Xorg + Steam

Posted at — May 03, 2023

Context

Summer is coming. Summer is hot. Gaming on desktop computer makes my office hotter. My laptop on the other hand doesn’t make it that much hotter.

My basement is cold. Therefore my Proxmox server is cold, and it doesn’t make my office hotter.

Idea

Put my video card in my Proxmox server and play remotely using Steam Remote Play.

Putting everything in a LXC container allows to keep things nice, tidy and separate from the host system as much as possible. The main goal for me is not security (you have to give the container access to a lot of things), but simply handling backups in a generic manner. It also allow me to snapshot the container easily before trying some dumb idea on it.

Now, starting a headless Steam that you can remote control is actually quite tricky. Lucky for us, someone already did the job for us: Headless Steam Service. Of course it means making Docker work in a LXC container, which is not that easy either.

Why not a VM with GPU passthrough ?

I tried and the frame rate was crap. Also it added some annoying latency that made FPS games harder to play (and much more frustrating).

Drawbacks

Security

The container has to be priviledged. I haven’t been able to make it work with unpriviledged containers.

Also, at the moment, the VNC server from Steam Headless Service doesn’t request any password at all …

Console

We have to pass the devices /dev/tty* through to the LXC container devices for Xorg to work. Unfortunately, this makes the container console in Proxmox unusable. It also means the container will reuse the terminal from the host system, which makes it quite unusable too.

Instructions

Host system

Nvidia drivers

You have to install nvidia-driver. Be careful, the very same version of the Nvidia libraries must be installed on the host system and in the container. If you use anything else than Debian, it may be tricky to get it right.

At the time of writing, AFAIK Steam requires nvidia drivers with version > 500, but Debian Bullseye (and therefore Proxmox) only provide drivers < 500. Also AFAIK Nvidia drivers and librairies go hand in hand regarding versions.

The easiest way to work around this problem is to install the nvidia drivers from the Nvidia Cuda repositories:

wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.0-1_all.deb
dpkg -i cuda-keyring_*deb
apt update
apt dist-upgrade
apt install nvidia-driver

# and also because otherwise it doesn't seem to work:
apt install curl sudo gpg
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
apt install nvidia-docker2 nvidia-container-runtime

Nvidia modules

Make sure the nvidia modules are loaded when the server start. Otherwise, from what I’ve seen, nvidia_uvm is not always loaded (loaded only on-demand ?)

cat <<EOF >>/etc/modules
nvidia
nvidia_uvm
EOF

Container

Proxmox LXC

Here is the configuration I use for my container:

arch: amd64
cores: 24
features: fuse=1,nesting=1
hostname: gaming
memory: 32768
net0: name=eth0,bridge=vmbr0,hwaddr=62:9E:8E:4D:2D:A5,ip=dhcp,ip6=auto,type=veth
onboot: 1
ostype: debian
rootfs: slowng:subvol-106-disk-0,size=512G
startup: order=60,up=20
swap: 512
mp0: /mnt/games,mp=/mnt/games,mountoptions=noatime
tty: 0
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-caps dev/nvidia-caps none bind,optional,create=dir
lxc.mount.entry: /dev/bus dev/bus none bind,optional,create=dir
lxc.mount.entry: /dev/input dev/input none bind,optional,create=dir
lxc.mount.entry: /dev/tty0 dev/tty0 none bind,optional,create=file
lxc.mount.entry: /dev/tty1 dev/tty1 none bind,optional,create=file
lxc.mount.entry: /dev/tty2 dev/tty2 none bind,optional,create=file
lxc.mount.entry: /dev/tty3 dev/tty3 none bind,optional,create=file
lxc.mount.entry: /dev/tty4 dev/tty4 none bind,optional,create=file
lxc.mount.entry: /dev/tty5 dev/tty5 none bind,optional,create=file
lxc.mount.entry: /dev/tty6 dev/tty6 none bind,optional,create=file
lxc.mount.entry: /dev/tty7 dev/tty7 none bind,optional,create=file
lxc.mount.entry: /dev/uinput dev/uinput none bind,optional,create=file
lxc.mount.entry: /dev/usb dev/usb none bind,optional,create=dir
lxc.mount.entry: /proc proc none bind,optional,create=dir
lxc.mount.entry: /sys sys none bind,optional,create=dir
# because everyone knows security is for chickens
lxc.cap.drop:
lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a

Regarding the lxc.cgroup2.devices.allow, you will have to adjust the values based on what ls -l /dev/nvidia* displays.

Since /dev/tty* are redirected, make sure you can access the container using SSH.

APT installing stuff

You need the very same Nvidia driver and librairies than your host:

wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.0-1_all.deb
dpkg -i cuda-keyring_*deb
apt update
apt dist-upgrade
apt install nvidia-driver

Then you need Docker:

apt install docker.io fuse-overlayfs

Docker + overlayfs + ZFS appears to work at first .. but then it really doesn’t (random errors when building images or running containers). So fuse-overlayfs is our friend here.

Then you need the Nvidia packages to make your Nvidia card available to stuff in Docker containers:

apt install curl sudo gpg

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt update
apt install nvidia-docker2 nvidia-container-runtime

Then you need docker-compose, but a version fresh enough so that it supports GPU passthrough. Unfortunately, it’s not the case with the docker-compose of Debian Bullseye (Update: docker-compose is Debian bookworm is OK), so:

apt install python3-pip
pip install docker-compose

Optionally:

adduser <your_login>
adduser <your_login> docker

Configuring Docker

We need to tell Docker to use fuse-overlayfs instead of overlayfs.

cat << EOF > /etc/docker/daemon.json
{
    "storage-driver": "fuse-overlayfs",
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
EOF

Also, you must set “no-cgroups = true” in the Nvidia docker config file /etc/nvidia-container-runtime/config.toml.

Installing Steam Headless

Here is the docker-compose.yml I use:

version: "3.8"
services:
  steam-headless:
    env_file:
      - env
      - passwd.env

    image: josh5/steam-headless:latest
    restart: "unless-stopped"
    # Steam Headless suggest specifying the nvidia runtime like that:
    # runtime: nvidia
    # but my solution (deploy: ...) is better :-P

    # because once again, security is for chickens
    privileged: true
    shm_size: 2G
    ipc: host # Could also be set to 'shareable'
    ulimits:
      nofile:
        soft: 1024
        hard: 524288

    ## NOTE:  Steam headless always requires the use of the host network.
    ##        If we do not use the host network, then device input is not possible
    ##        and your controllers will not work in steam games.
    network_mode: host
    hostname: SteamGaia
    extra_hosts:
      - "SteamGaia:127.0.0.1"

    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

    volumes:
      # The location of your home directory.
      - /mnt/games/steam/home:/home/default/:rw
      # The location where all games should be installed.
      - /mnt/games/steam/games:/mnt/games/:rw
      # Input devices used for mouse and joypad support inside the container.
      # (I don't use it, but why not)
      - /dev/input/:/dev/input/:ro
      # The Xorg socket. This will be shared with other containers so they can access the X server.
      - /mnt/games/steam/steam-headless/.X11-unix/:/tmp/.X11-unix/:rw
      # Pulse audio socket. This will be shared with other containers so they can access the audio sink.
      - /mnt/games/steam/steam-headless/pulse/:/tmp/pulse/:rw

With the file env containing:

NAME='SteamGaia'
TZ='Europe/Paris'
USER_LOCALES='en_US.UTF-8 UTF-8'
DISPLAY=':55'
SHM_SIZE='2G'

# DOCKER_RUNTIME='runc'
DOCKER_RUNTIME='nvidia'

PUID='1000'
PGID='1000'
UMASK='000'
# USER_PASSWORD='password'

MODE='primary'

WEB_UI_MODE='vnc'
ENABLE_VNC_AUDIO='true'
PORT_NOVNC_WEB='8083'
NEKO_NAT1TO1=''
ENABLE_SUNSHINE='false'
#SUNSHINE_USER='admin'
#SUNSHINE_PASS='admin'

ENABLE_EVDEV_INPUTS='true'
NVIDIA_DRIVER_CAPABILITIES='all'
NVIDIA_VISIBLE_DEVICES='all'

Then:

echo 'USER_PASSWORD="your_password"' > passwd.env

And then:

docker-compose up -d

And maybe, if you feel like it:

docker-compose logs -f

Connecting to the Steam Headless

I for one use the VNC client Remmina.

You can simply tell it to connect to <your_container>:32036. Beware, sometimes, randomly, it decides to use the port 32037 instead of 32036.

Once you have installed your games, you can simply disconnect from VNC and use Steam Remote Play to enjoy your games remotely.

Conclusion

I problably forgot some instructions … so .. happy debugging :-D