Skip to content

Proxmox host GPU passthrough

Enable IOMMU in the BIOS

1. Host BIOS

For each CPU type ensure that the following is enabled

  • Intel: Enable VT-d in BIOS.
  • AMD: Enable IOMMU in BIOS.

For SuperMicro you will need to enable Extended APIC as well as Intel Virtualization Technology, which can be found in BIOS -> Advanced -> CPU Configuration

2. Update GRUB Configuration

Open the GRUB configuration file:

Edit grub

Modify the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub to enable IOMMU:

# For Intel processors
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
# OR
# For AMD processors
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

Update Grub

Now, the grub bootloader must be updated to the new config.

update-grub

*Requires a reboot to be applied, but you can wait for the reboot at the end of the installation

Blacklist kernel drivers on host

1. Check currently loaded kernel drivers

This command will list all detected nvdia GPUs on the system.

lspci -vvv | grep -i nvidia
Take note of the Kernel Modules that are configured for each GPU device

Example response

51:00.0 VGA compatible controller: NVIDIA Corporation GA104GL [RTX A4000] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation GA104GL [RTX A4000]
        Kernel modules: nvidiafb, nouveau
51:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation GA104 High Definition Audio Controller
8a:00.0 VGA compatible controller: NVIDIA Corporation GA104GL [RTX A4000] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation GA104GL [RTX A4000]
        Kernel modules: nvidiafb, nouveau
8a:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation GA104 High Definition Audio Controller
c3:00.0 VGA compatible controller: NVIDIA Corporation GA104GL [RTX A4000] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation GA104GL [RTX A4000]
        Kernel modules: nvidiafb, nouveau
c3:00.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation GA104 High Definition Audio Controller
Thus the first A4000 GPU has - Kernel Modules: nvidiafb, nouveau

2. blacklist.conf

Disable the kernel modules/drivers from being consumed by the Proxmox host on boot

Create or edit the file /etc/modprobe.d/blacklist.conf and add the contents

blacklist nouveau
blacklist nvidiafb
blacklist nvidia

Configure device passthough

1. Get device IDs

Run the below command to identify the device id's to be passed through to the virtualisation services. 

lspci -nn | grep -i nvidia
Take note of the device ids

Example response

51:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1)
51:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
8a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1)
8a:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
c3:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1)
c3:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
Thus the first A4000 GPU has - IDs: 10de:24b0, 10de:228b

Not the ID at the beginning of the line

2. vfio.conf

Now configure the nvidia kernel modules and devices are passed through to the porxmox virtualisation driver.

Create or edit the file /etc/modprobe.d/vfio.conf and add the contents.

The vfio-pci ids are a comma seperated list of those identified in the previous step

softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidiafb pre: vfio-pci
softdep nvidia_drm pre: vfio-pci
softdep drm pre: vfio-pci
options vfio-pci ids=10de:24b0,10de:228b

Save and Restart host

Commit config and restart the proxmox host to reload the kernel drivers.

Update the initramfs after all changes

update-initramfs -u
#update-initramfs -u -k all

# Response
update-initramfs: Generating /boot/initrd.img-6.5.11-8-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

reboot

Confirmation

Check for drivers

lsmod | grep -E 'nouveau|nvidia'
# No output means it's good.

Check for in use kernel driver

Look at Kernel driver in use: vfio-pci. If that is not present, then there is something wrong with the config

lspci -nnk -d 10de:
# Example Response
51:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1)
        Subsystem: NVIDIA Corporation GA104GL [RTX A4000] [10de:14ad]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
51:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
        Subsystem: NVIDIA Corporation GA104 High Definition Audio Controller [10de:14ad]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
8a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1)
        Subsystem: NVIDIA Corporation GA104GL [RTX A4000] [10de:14ad]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
8a:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
        Subsystem: NVIDIA Corporation GA104 High Definition Audio Controller [10de:14ad]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
c3:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104GL [RTX A4000] [10de:24b0] (rev a1)
        Subsystem: NVIDIA Corporation GA104GL [RTX A4000] [10de:14ad]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
c3:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
        Subsystem: NVIDIA Corporation GA104 High Definition Audio Controller [10de:14ad]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

Check the grub commandline

If the grub commandline does not contain the iommu entries (provided when you edited grub), you will need to check step 1 (and reboot afterward) 

cat /proc/cmdline

Next Steps

You should, now, be able to deploy VMs with GPUs.

Go here