Set up NVIDIA GPU passthrough

Reserve an NVIDIA card for VFIO on a Proxmox VE 9 host in one paste — module binding, initramfs, and the kernel command line for whichever bootloader the host uses.

Overview

This runbook explains how to prepare a Proxmox VE 9 host to pass an NVIDIA GPU through to a guest.

Afterwards the card you name is claimed by vfio-pci at boot instead of by nouveau, the VFIO modules are in the initramfs and loaded on the running system, and the kernel command line carries whatever the card needs. It stops there: assigning the card to a VM is a per-guest hostpci setting, and the Proxmox wiki covers it.

Proxmox already ships QEMU and its own VM management, so nothing is installed here and no repository is touched. This is entirely host configuration, and none of it takes effect until the machine reboots.

Written against Proxmox VE 9.2 — Debian 13.5, Linux 7.0 — and it holds across 9.x. It does not hold on Proxmox VE 8: the IOMMU is only on by default for Intel CPUs from Linux 6.8, and 8.x shipped older kernels that still needed intel_iommu=on.

Before you start

Before you run this, ensure:

The card is not needed by the host for anything. Proxmox does not use a GPU itself, so on most hosts this is free.
You can reach the host without its console — the web UI on port 8006, or SSH. If the card you pass through is the one the host boots on, its local display goes away permanently at the next boot, and the script makes you type yes before it will do that.
Virtualisation and the IOMMU are on in firmware: SVM Mode and IOMMU on AMD, VT-x and VT-d on Intel. The script stops if either is missing, and on Proxmox VE 9 no kernel argument can substitute for the firmware setting.
You can afford a reboot, now or soon. Until then the configuration is written but the card has not moved.
Any VM that already has this card assigned is shut down.

The script asks you for:

GPU_SLOT — the PCI slot of the card, as 0000:01:00. The script lists every NVIDIA display and 3D controller it finds, marks the one the host boots on, and defaults to the first one that is not.
A typed yes, and only if the card you chose is the one driving the host console.
WANT_PT — whether to add iommu=pt to the kernel command line. Recommended, and the default.
WANT_REBOOT — whether to reboot at the end.

The script

Every check runs before anything is written, so a host that cannot do this is turned away unchanged.

bash

(
  set -euo pipefail

  command -v pveversion >/dev/null ||
    { echo "This is not a Proxmox VE host." >&2; exit 1; }

  # ---- Everything it asks for. Nothing here changes the machine. ----

  mapfile -t NVIDIA_SLOTS < <(
    { lspci -D -d '10de::0300'; lspci -D -d '10de::0302'; } |
      awk '{ sub(/\.[0-9]+$/, "", $1); print $1 }' | sort -u
  )
  if [[ ${#NVIDIA_SLOTS[@]} -eq 0 ]]; then
    echo "No NVIDIA display or 3D controller on this host." >&2
    exit 1
  fi

  DEFAULT_SLOT=""
  echo "NVIDIA cards on this host:"
  for slot in "${NVIDIA_SLOTS[@]}"; do
    desc=$(lspci -nn -s "$slot.0" | cut -d' ' -f2-)
    if [[ "$(cat "/sys/bus/pci/devices/$slot.0/boot_vga" 2>/dev/null)" == "1" ]]; then
      echo "  $slot  $desc  [the host boots on this one]"
    else
      echo "  $slot  $desc"
      DEFAULT_SLOT=${DEFAULT_SLOT:-$slot}
    fi
  done

  read -rp "GPU to pass through${DEFAULT_SLOT:+ [$DEFAULT_SLOT]}: " GPU_SLOT
  GPU_SLOT=${GPU_SLOT:-$DEFAULT_SLOT}
  [[ -n "$GPU_SLOT" ]] || { echo "No card chosen." >&2; exit 1; }
  [[ "$GPU_SLOT" == *:*:* ]] || GPU_SLOT="0000:$GPU_SLOT"
  [[ -e "/sys/bus/pci/devices/$GPU_SLOT.0" ]] ||
    { echo "No PCI device at $GPU_SLOT.0" >&2; exit 1; }

  PRIMARY=0
  if [[ "$(cat "/sys/bus/pci/devices/$GPU_SLOT.0/boot_vga" 2>/dev/null)" == "1" ]]; then
    PRIMARY=1
    echo
    echo "$GPU_SLOT is the card this host boots on and prints its console to."
    echo "Handing it to a guest takes that away: after the next boot this"
    echo "machine has no local display at all, and the only way in is the web"
    echo "UI on port 8006 or SSH. Undoing it means editing the kernel command"
    echo "line, which without a console means a rescue medium."
    read -rp "Type 'yes' to pass through the console GPU anyway: " CONFIRM
    [[ "$CONFIRM" == "yes" ]] || { echo "Nothing done." >&2; exit 1; }
  fi

  read -rp "Add iommu=pt to the kernel command line? [Y/n]: " WANT_PT
  read -rp "Reboot when the script finishes? [y/N]: " WANT_REBOOT

  SUDO="sudo"
  if [[ "$EUID" -eq 0 ]]; then
    SUDO=""
  elif ! sudo -v; then
    echo "Run this as root." >&2
    exit 1
  fi

  # ---- Checks. Still nothing changed. ----

  echo "==> Checking the host can do this"
  echo "    $(pveversion)"

  grep -qw -e vmx -e svm /proc/cpuinfo ||
    { echo "No vmx/svm flag: virtualisation is off in firmware." >&2; exit 1; }

  if ! compgen -G '/sys/class/iommu/*' >/dev/null; then
    echo "The IOMMU is not active. On Proxmox VE 9 the kernel turns it on by" >&2
    echo "itself when the hardware has it, so this means it is off in" >&2
    echo "firmware: enable VT-d or AMD-Vi there and run this again." >&2
    exit 1
  fi

  [[ -f /etc/modules ]] ||
    { echo "No /etc/modules on this host." >&2; exit 1; }
  [[ -f /etc/initramfs-tools/modules ]] ||
    { echo "No /etc/initramfs-tools/modules; this host does not build its" >&2
      echo "initramfs with initramfs-tools." >&2; exit 1; }

  mapfile -t GPU_IDS < <(lspci -n -s "$GPU_SLOT." | awk '{ print $3 }')
  mapfile -t GPU_MODULES < <(
    lspci -nnk -s "$GPU_SLOT." |
      sed -n 's/^[[:space:]]*Kernel modules:[[:space:]]*//p' |
      tr ',' '\n' | sed 's/[[:space:]]//g' | grep -v '^$' | sort -u
  )
  [[ ${#GPU_IDS[@]} -gt 0 ]] ||
    { echo "Found no PCI functions at $GPU_SLOT." >&2; exit 1; }
  GPU_ID_LIST=$(IFS=,; echo "${GPU_IDS[*]}")

  # A bridge or root port sharing the group is expected and harmless; any
  # other device in it is not, because VFIO hands a group over whole.
  mapfile -t STRANGERS < <(
    for dev in "/sys/bus/pci/devices/$GPU_SLOT.0/iommu_group/devices"/*; do
      base=${dev##*/}
      [[ "$base" == "$GPU_SLOT."* ]] && continue
      class=$(cat "/sys/bus/pci/devices/$base/class")
      [[ "$class" == 0x0604* || "$class" == 0x0600* ]] && continue
      echo "$base $(lspci -nn -s "$base" | cut -d' ' -f2-)"
    done
  )
  if [[ ${#STRANGERS[@]} -gt 0 ]]; then
    echo "$GPU_SLOT shares its IOMMU group with devices that are not bridges:" >&2
    printf '  %s\n' "${STRANGERS[@]}" >&2
    echo "VFIO hands a group over whole or not at all, so those would have to" >&2
    echo "go to the guest too. Move the card to another PCIe slot." >&2
    exit 1
  fi

  EXTRA_ARGS=""
  [[ "$WANT_PT" == [Nn]* ]] || EXTRA_ARGS="iommu=pt"
  if [[ "$PRIMARY" -eq 1 ]]; then
    EXTRA_ARGS="${EXTRA_ARGS:+$EXTRA_ARGS }initcall_blacklist=sysfb_init"
  fi

  echo "    ${#GPU_IDS[@]} function(s) at $GPU_SLOT: $GPU_ID_LIST"
  echo "    drivers to out-rank: ${GPU_MODULES[*]}"
  echo "    kernel arguments to add: ${EXTRA_ARGS:-none}"

  # ---- Everything below here changes the machine. ----

  echo "==> Reserving $GPU_SLOT for vfio-pci"
  {
    echo "# NVIDIA GPU at $GPU_SLOT, reserved for VFIO passthrough."
    echo "# Written by the Proxmox VE 9 GPU passthrough runbook."
    echo "options vfio-pci ids=$GPU_ID_LIST"
    for mod in "${GPU_MODULES[@]}"; do
      echo "softdep $mod pre: vfio-pci"
    done
  } | $SUDO tee /etc/modprobe.d/vfio.conf >/dev/null

  echo "==> Loading the VFIO modules at boot and in the initramfs"
  for file in /etc/modules /etc/initramfs-tools/modules; do
    for mod in vfio vfio_iommu_type1 vfio_pci; do
      if ! grep -qxF "$mod" "$file"; then
        echo "$mod" | $SUDO tee -a "$file" >/dev/null
      fi
    done
  done

  strip_args() {
    printf '%s' "$1" | tr ' ' '\n' |
      grep -vE '^(iommu=pt|initcall_blacklist=sysfb_init)$' | paste -sd' ' - || true
  }

  if [[ -n "$EXTRA_ARGS" ]]; then
    echo "==> Putting $EXTRA_ARGS on the kernel command line"
    TOUCHED_BOOT=0

    if [[ -f /etc/kernel/cmdline ]]; then
      $SUDO cp -n /etc/kernel/cmdline /etc/kernel/cmdline.dist
      CUR=$(tr '\n' ' ' < /etc/kernel/cmdline)
      NEW=$(strip_args "$CUR")
      printf '%s\n' "${NEW:+$NEW }$EXTRA_ARGS" |
        $SUDO tee /etc/kernel/cmdline >/dev/null
      echo "    /etc/kernel/cmdline: $(cat /etc/kernel/cmdline)"
      TOUCHED_BOOT=1
    fi

    if [[ -f /etc/default/grub && -f /boot/grub/grub.cfg ]]; then
      $SUDO cp -n /etc/default/grub /etc/default/grub.dist
      CUR=$( . /etc/default/grub; echo "${GRUB_CMDLINE_LINUX_DEFAULT:-}" )
      NEW=$(strip_args "$CUR")
      NEW="${NEW:+$NEW }$EXTRA_ARGS"
      if grep -q '^GRUB_CMDLINE_LINUX_DEFAULT=' /etc/default/grub; then
        $SUDO sed -i \
          "s|^GRUB_CMDLINE_LINUX_DEFAULT=.*|GRUB_CMDLINE_LINUX_DEFAULT=\"$NEW\"|" \
          /etc/default/grub
      else
        echo "GRUB_CMDLINE_LINUX_DEFAULT=\"$NEW\"" |
          $SUDO tee -a /etc/default/grub >/dev/null
      fi
      echo "    /etc/default/grub: $NEW"
      TOUCHED_BOOT=1
    fi

    if [[ "$TOUCHED_BOOT" -eq 0 ]]; then
      echo "Found neither /etc/kernel/cmdline nor a GRUB configuration, so" >&2
      echo "there is nowhere to put $EXTRA_ARGS." >&2
      exit 1
    fi
  fi

  echo "==> Rebuilding the initramfs and refreshing the bootloader"
  $SUDO update-initramfs -u -k all
  if [[ -f /boot/grub/grub.cfg ]]; then
    $SUDO update-grub
  fi
  if $SUDO proxmox-boot-tool status >/dev/null 2>&1; then
    $SUDO proxmox-boot-tool refresh
  fi

  NODE=$(hostname)

  echo
  echo "Done, but not in effect: the running kernel still has $GPU_SLOT on"
  echo "its host driver. The card moves at the next boot."
  echo "  vfio-pci will claim ids=$GPU_ID_LIST"
  if [[ "$PRIMARY" -eq 1 ]]; then
    echo "  This host will have no local console after the reboot."
  fi
  echo
  echo "After the reboot, check:"
  echo "  lspci -nnk -s $GPU_SLOT.   # every function must say vfio-pci"
  echo "  lsmod | grep vfio"
  echo "  pvesh get /nodes/$NODE/hardware/pci --pci-class-blacklist ''"

  if [[ "$WANT_REBOOT" == [Yy]* ]]; then
    echo
    echo "Rebooting in 5 seconds. Ctrl-C to stay up."
    sleep 5
    $SUDO systemctl reboot
  fi
)

How it works

Choosing the card

lspci -D -d '10de::0300' and -d '10de::0302' list PCI devices whose vendor is NVIDIA — 10de — and whose class is VGA compatible controller or 3D controller. Compute cards present as the latter and have no VGA class at all, which is why both are asked for. -D prints the PCI domain, so the slots come out as 0000:01:00 and every sysfs path below can be built from them. The slot you type is normalised to add a domain if you left it out.

boot_vga in sysfs is the kernel's record of which adapter the firmware booted with. On a Proxmox host that is often the only adapter, which is why this is a typed confirmation rather than a refusal: a headless hypervisor administered through its web UI has no real use for a local console, and passing its single GPU to a guest is a perfectly ordinary thing to do. It is also the one choice here you cannot walk back without physical access, so it asks.

What is not checked, and why

There is no intel_iommu=on anywhere in this script. From Linux 6.8 the kernel enables the IOMMU by default on Intel as well as AMD, and Proxmox VE 9 has never shipped a kernel older than that. So an empty /sys/class/iommu/ on this host does not mean a missing kernel argument — it means VT-d or AMD-Vi is switched off in firmware, and no amount of configuration will help. The script says so and stops, rather than writing an argument that cannot fix it.

vmx or svm in /proc/cpuinfo is checked for the same reason: it is what the CPU reports when hardware virtualisation is on in firmware.

Reading the card's configuration off the card

lspci -n -s "$GPU_SLOT." — note the trailing dot, which selects every function of the slot rather than function zero — gives the vendor:device pair for each. A graphics card is at least two functions, the display controller and an HDMI/DisplayPort audio controller, and both go to the guest. Kernel modules: from lspci -k names every driver that could claim each function, and each becomes a softdep. Both lists are read off the running host, so nothing in this script needs editing for your hardware, and a host that has the proprietary nvidia driver installed gets softdep nvidia pre: vfio-pci without anyone remembering it.

Checking the card is on its own

An IOMMU group is the smallest set of devices that can be isolated from each other, and VFIO hands a group over whole or not at all. If an NVMe drive or a USB controller shares the group, it would have to go to the guest too.

PCI bridges and root ports are the exception, and the script skips them by class code — 0x0604 for a PCI-to-PCI bridge, 0x0600 for a host bridge. They routinely sit in a group with the devices behind them and are not themselves assigned to anything, so treating them as a conflict would reject almost every working configuration. Anything else in the group stops the run, with the offending devices named. The answer is a different PCIe slot, not an ACS override patch, which fakes the isolation the group exists to describe.

Reserving the card

options vfio-pci ids=… tells vfio-pci which devices to claim when it loads. On its own that is not enough, because whichever driver reaches a device first keeps it: snd_hda_intel loads early for the host's onboard audio and takes the GPU's audio function on the way past, leaving the display controller on vfio-pci and the audio function on snd_hda_intel. Proxmox then tries to take that function back when the guest starts and fails, because the host has the device open — so the passthrough works until the day it does not. The softdep … pre: vfio-pci lines are what prevent that.

softdep is used rather than the blanket blacklist nouveau that most guides reach for, because it is targeted: vfio-pci only claims the IDs it was given, so a second NVIDIA card the host does keep carries on working normally. If a function still reports its old driver after the reboot, the softdep name is wrong — it has to match the Kernel modules: line exactly, underscores and all — and blacklisting that driver outright is the fallback.

The file has to be named *.conf: mkinitramfs copies /etc/modprobe.d/*.conf into the initramfs and ignores anything else, and a vfio file without the extension would be read on the running system but not during early boot, which is exactly where the race is decided.

Two module lists, two jobs

/etc/modules and /etc/initramfs-tools/modules get the same three entries, and it is not redundant. /etc/modules is read at boot on the real root, which is what the Proxmox wiki asks for. /etc/initramfs-tools/modules is the only one mkinitramfs reads — it does not look at /etc/modules at all — and it is what puts vfio-pci inside the initramfs, where it has to exist before a softdep can prefer it over a driver loading there. Both are appended to only when the entry is absent, so pasting this twice does not accumulate duplicates.

The kernel command line, on both bootloaders

Which file holds the kernel command line depends on how the host was installed: /etc/kernel/cmdline for systemd-boot, which a ZFS root on UEFI without Secure Boot gets, and GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub for everything else. proxmox-boot-tool status can also report a single ESP configured with both, so rather than guess, the script writes to each file that is present and applicable and lets the one that runs pick it up. Writing the argument twice costs nothing; writing it to the wrong one costs a reboot to discover.

/etc/default/grub is sourced rather than pattern-matched, so the value read is the one GRUB itself would use — the last assignment wins and a commented-out line above it is ignored, both of which a sed would get wrong. In both files the arguments this script manages are stripped before being re-added, which is what makes a second run idempotent instead of leaving iommu=pt iommu=pt behind. Each file is copied to a .dist sibling on the first run, with cp -n, so the copy stays the pristine original however many times you paste this.

iommu=pt is passthrough mode. It leaves devices the host still owns on the untranslated DMA path, which is faster, while devices actually assigned to VFIO still get full translation. It is a performance option, not a correctness one, which is why it is a question rather than a given.

initcall_blacklist=sysfb_init is added only when you confirmed the card is the one the host boots on. The firmware framebuffer set up during boot keeps a claim on the card's memory BARs, and VFIO cannot then reserve them — the failure shows up as BAR 0: can't reserve in the log and a guest that will not start. Blocking that initcall stops the framebuffer being set up at all, at the cost of the host printing nothing to the screen from early boot onwards. This is the argument that replaced the older video=efifb:off advice, which has done nothing on these kernels for several releases.

Rebuilding

update-initramfs -u -k all rebuilds every installed kernel's initramfs, not just the running one, so the card still lands on vfio-pci if you boot an older kernel from the boot menu. update-grub writes the new command line into /boot/grub/grub.cfg, and is run only where that file already exists, so it is skipped on a host that boots with systemd-boot alone.

proxmox-boot-tool refresh copies the kernels, initramfs images, and boot entries onto every ESP the tool manages. update-initramfs already triggers a refresh on those hosts, so this is belt and braces rather than a missing step — but it is cheap, and it is the difference between a configuration that is written and one that is on the disk the firmware actually reads.

Nothing takes effect until the reboot. To undo all of it: delete /etc/modprobe.d/vfio.conf, restore the .dist copies of whichever command line files were touched, remove the three vfio lines from both module lists, and run the same rebuild commands again.