Proxmox LXC NVIDIA
Original posted at jocke.no
Also with information from theorangeone.net
Proxmox Host
First step is to install the drivers on the host. Nvidia has an official Debian repo, that we could use. However, that introduces a potential problem; we need to install the drivers on the LXC container later without kernel modules. I could not find a way to do this using the packages within the official Debian repo, and therefore had to install the drivers manually within the LXC container. The other aspect is that both the host and the LXC container need to run the same driver version (or else it won’t work). If we install using official Debian repo on the host, and manual driver install on the LXC container, we could easily end up with different versions (whenever you do an apt upgrade on the host). In order to have this as consistent as possible, we’ll install the driver manually on both the host and within the LXC container.
echo -e "blacklist nouveau\noptions nouveau modeset=0" > /etc/modprobe.d/blacklist-nouveau.conf
update-initramfs -u
reboot
Install pve headers matching your current kernel
apt install pve-headers-$(uname -r)
Download + install nvidia driver
Note: 520.56.06 was the latest at the time of this writing, however check https://download.nvidia.com/XFree86/Linux-x86_64/ for newer drivers
Answer "no" when it asks if you want to install 32bit compability drivers
Answer "no" when it asks if it should update X config
wget -O NVIDIA-Linux-x86_64.run https://download.nvidia.com/XFree86/Linux-x86_64/520.56.06/NVIDIA-Linux-x86_64-520.56.06.run
chmod +x NVIDIA-Linux-x86_64.run
./NVIDIA-Linux-x86_64.run --check
./NVIDIA-Linux-x86_64.run
With the drivers installed, we need to add some udev-rules. This is to make sure proper kernel modules are loaded, and that all the relevant device files is created upon boot.
Add kernel modules
echo -e '\n# load nvidia modules\nnvidia\nnvidia_uvm\nnvidia-drm\nnvidia-uvm' >> /etc/modules-load.d/modules.conf
Once that’s done, you’ll need to update the initramfs
update-initramfs -u -k all.
Add the following to /etc/udev/rules.d/70-nvidia.rules Will create relevant device files within /dev/ during boot
nano /etc/udev/rules.d/70-nvidia.rules
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"
SUBSYSTEM=="module", ACTION=="add", DEVPATH=="/module/nvidia", RUN+="/usr/bin/nvidia-modprobe -m"
To avoid that the driver/kernel module is unloaded whenever the GPU is not used, we should run the Nvidia provided persistence service. It’s made available to us after the driver install.
Copy and extract
cp /usr/share/doc/NVIDIA_GLX-1.0/samples/nvidia-persistenced-init.tar.bz2 .
bunzip2 nvidia-persistenced-init.tar.bz2
tar -xf nvidia-persistenced-init.tar
Remove old, if any (to avoid masked service)
rm /etc/systemd/system/nvidia-persistenced.service
Install
chmod +x nvidia-persistenced-init/install.sh
./nvidia-persistenced-init/install.sh
Check that it's ok
systemctl status nvidia-persistenced.service
rm -rf nvidia-persistenced-init*
If you’ve come so far without any errors, you’re ready to reboot the Proxmox host. After the reboot, you should see the following outputs (GPU type/info will of course change depending on your GPU);
nvidia-smi
Output
Wed Feb 23 01:34:17 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA RTX A2000 On | 00000000:82:00.0 Off | Off |
| 30% 36C P2 4W / 70W | 1MiB / 6138MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
systemctl status nvidia-persistenced.service
Output
● nvidia-persistenced.service - NVIDIA Persistence Daemon
Loaded: loaded (/lib/systemd/system/nvidia-persistenced.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-02-23 00:18:04 CET; 1h 16min ago
Process: 9300 ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced (code=exited, status=0/SUCCESS)
Main PID: 9306 (nvidia-persiste)
Tasks: 1 (limit: 154511)
Memory: 512.0K
CPU: 1.309s
CGroup: /system.slice/nvidia-persistenced.service
└─9306 /usr/bin/nvidia-persistenced --user nvidia-persistenced
Feb 23 00:18:03 foobar systemd[1]: Starting NVIDIA Persistence Daemon...
Feb 23 00:18:03 foobar nvidia-persistenced[9306]: Started (9306)
Feb 23 00:18:04 foobar systemd[1]: Started NVIDIA Persistence Daemon.
root@foobar:~# ls -alh /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 Feb 23 00:17 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb 23 00:17 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Feb 23 00:17 /dev/nvidia-modeset
crw-rw-rw- 1 root root 511, 0 Feb 23 00:17 /dev/nvidia-uvm
crw-rw-rw- 1 root root 511, 1 Feb 23 00:17 /dev/nvidia-uvm-tools
If the correct GPU shows from nvidia-smi, the persistence service runs fine, and all five files are available, we’re ready to proceed to the LXC container.
LXC container
We need to add relevant LXC configuration to our container. Shut down the LXC container, and make the following changes to the LXC configuration file;
Edit /etc/pve/lxc/1xx.conf and add the following
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 509:* rwm
lxc.cgroup2.devices.allow: c 511:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
The numbers on the cgroup2-lines are from the fifth column in the device-list above (via ls -alh /dev/nvidia*). For me, the two nvidia-uvm files changes randomly between 509 and 511, while the three others remain static as 195. I don’t know why they alternate between the two values (if you know how to make them static, please let me know), but LXC does not complain if you configure numbers that doesn’t exist (i.e. we can add all three of them to make sure it works).
We can now turn on the LXC container, and we’ll be ready to install the Nvidia driver. This time we’re going to install it without the kernel drivers, and there is no need to install the kernel headers.
Answer "no" when it asks if it should update X config
wget -O NVIDIA-Linux-x86_64.run https://download.nvidia.com/XFree86/Linux-x86_64/520.56.06/NVIDIA-Linux-x86_64-520.56.06.run
chmod +x NVIDIA-Linux-x86_64.run
./NVIDIA-Linux-x86_64.run --check
./NVIDIA-Linux-x86_64.run --no-kernel-module
At this point you should be able to reboot your LXC container. Verify that the files and driver works as expected, before moving on to the Docker setup.
ls -alh /dev/nvidia*
Output
crw-rw-rw- 1 root root 195, 0 Feb 23 00:17 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb 23 00:17 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Feb 23 00:17 /dev/nvidia-modeset
crw-rw-rw- 1 root root 511, 0 Feb 23 00:17 /dev/nvidia-uvm
crw-rw-rw- 1 root root 511, 1 Feb 23 00:17 /dev/nvidia-uvm-tools
run nvidia-smi
nvidia-smi
Output
Wed Feb 23 01:50:15 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA RTX A2000 Off | 00000000:82:00.0 Off | Off |
| 30% 34C P8 10W / 70W | 3MiB / 6138MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Docker container
Now we can move on to get the Docker working. We’ll be using docker-compose, and we’ll also make sure to have the latest version by removing the Debian-provided docker and docker-compose. We’ll also install the Nvidia-provided Docker runtime. Both these are relevant in terms of making the GPU available within Docker.
Remove debian-provided packages
apt remove docker-compose docker docker.io containerd runc
Install docker from official repository
apt update
apt install ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
apt install docker-ce docker-ce-cli containerd.io
Install docker-compose
curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
Install docker-compose bash completion
curl \
-L https://raw.githubusercontent.com/docker/compose/1.29.2/contrib/completion/bash/docker-compose \
-o /etc/bash_completion.d/docker-compose
Install nvidia-docker2
apt install -y curl
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
keyring_file="/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg"
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o ${keyring_file}
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed "s#deb https://#deb [signed-by=${keyring_file}] https://#g" | \
tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
apt install nvidia-docker2
Restart systemd + docker (if you don't reload systemd, it might not work)
systemctl daemon-reload
systemctl restart docker
We should now be able to run Docker containers with GPU support. Let’s test it.
docker run --rm --gpus all nvidia/cuda:11.8.0-devel-ubuntu22.04 nvidia-smi
Output
Tue Feb 22 22:15:14 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06 Driver Version: 520.56.06 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA RTX A2000 Off | 00000000:82:00.0 Off | Off |
| 30% 29C P8 4W / 70W | 1MiB / 6138MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Testing Everything Works
An easy way to test it all works is to add a FileFlows container
nano docker-compose.yml
version: '3.7'
services:
fileflows:
image: revenz/fileflows
container_name: fileflows
runtime: nvidia
stdin_open: true # docker run -i
tty: true # docker run -t
environment:
- TZ=Pacific/Auckland
- TempPathHost=/temp
- NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
- NVIDIA_VISIBLE_DEVICES=all
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /mnt/docker/fileflows/data:/app/Data
- /mnt/docker/fileflows/logs:/app/Logs
- /mnt/docker/fileflows/temp:/temp
ports:
- 19200:5000
restart: unless-stopped
Run the container
docker-compose up
Then run a ffmpeg nvidia test
ffmpeg -loglevel error -f lavfi -i color=black:s=1920x1080 -vframes 1 -an -c:v hevc_nvenc -f null -
If the command completes without error, everything is working!