Set up NVIDIA GPU passthrough

Install the libvirt/QEMU stack on a Debian 13 host and reserve an NVIDIA card for VFIO in one paste, up to the reboot that hands the card over.

Overview

This runbook explains how to prepare a Debian 13 (trixie) host to pass an NVIDIA GPU through to a virtual machine.

Afterwards the host runs libvirt and QEMU with UEFI firmware available, the account you name can manage VMs, and the card you name is claimed by vfio-pci at boot instead of by nouveau or the NVIDIA driver. It stops there. Creating the VM and attaching the card to it is the next job, and Pass a GPU through to a VM covers it.

Nothing takes effect until the machine reboots — the running kernel keeps the card on whatever driver already has it. The script offers to reboot for you at the end.

The same commands hold on any x64 Debian 13 host with an AMD or Intel CPU. They assume GRUB, which is what the Debian installer sets up.

Before you start

Before you run this, ensure:

The host has a second display adapter — an integrated GPU or a spare card — driving its console, and firmware is set to boot from that one. A card handed to a VM stops being available to the host. The script refuses to touch a card that is currently driving the console.
Virtualisation and the IOMMU are enabled in firmware: SVM Mode and IOMMU on AMD, VT-x and VT-d on Intel. The script stops if the CPU reports no virtualisation support.
You have a way back in that does not depend on the card — SSH, or the console on the other adapter. See First boot.
You can reach the host on TCP port 9090 from a browser, if you want the Cockpit console.
You can afford a reboot, now or soon. The host is in a half-configured state until then: the configuration is written but the card has not moved.

The script asks you for:

GPU_SLOT — the PCI slot of the card to pass through, as 0000:01:00. The script lists every NVIDIA display and 3D controller it finds, marks the one driving the console, and defaults to the first one that is not.
VM_USER — the account to put in the libvirt group. Defaults to whoever invoked the script.
WANT_COCKPIT — whether to install the Cockpit web console and its Virtual machines page alongside the stack.
WANT_REBOOT — whether to reboot at the end, or leave that to you.

The script

It runs its checks before installing anything, so a host that cannot do this is turned away unchanged.

bash

(
  set -euo pipefail

  command -v lspci >/dev/null ||
    { echo "lspci is missing: run 'sudo apt install pciutils' first." >&2; exit 1; }

  # ---- Everything it asks for. Nothing here changes the machine. ----

  mapfile -t NVIDIA_SLOTS < <(
    { lspci -D -d '10de::0300'; lspci -D -d '10de::0302'; } |
      awk '{ sub(/\.[0-9]+$/, "", $1); print $1 }' | sort -u
  )
  if [[ ${#NVIDIA_SLOTS[@]} -eq 0 ]]; then
    echo "No NVIDIA display or 3D controller on this host." >&2
    exit 1
  fi

  DEFAULT_SLOT=""
  echo "NVIDIA cards on this host:"
  for slot in "${NVIDIA_SLOTS[@]}"; do
    if [[ "$(cat "/sys/bus/pci/devices/$slot.0/boot_vga" 2>/dev/null)" == "1" ]]; then
      echo "  $slot  $(lspci -nn -s "$slot.0" | cut -d' ' -f2-)  [drives the console]"
    else
      echo "  $slot  $(lspci -nn -s "$slot.0" | cut -d' ' -f2-)"
      DEFAULT_SLOT=${DEFAULT_SLOT:-$slot}
    fi
  done
  if [[ -z "$DEFAULT_SLOT" ]]; then
    echo "Every NVIDIA card here is driving the host console. Set the" >&2
    echo "integrated GPU as the primary display in firmware, reboot, and" >&2
    echo "run this again." >&2
    exit 1
  fi

  read -rp "GPU to pass through [$DEFAULT_SLOT]: " GPU_SLOT
  GPU_SLOT=${GPU_SLOT:-$DEFAULT_SLOT}
  [[ "$GPU_SLOT" == *:*:* ]] || GPU_SLOT="0000:$GPU_SLOT"
  [[ -e "/sys/bus/pci/devices/$GPU_SLOT.0" ]] ||
    { echo "No PCI device at $GPU_SLOT.0" >&2; exit 1; }
  if [[ "$(cat "/sys/bus/pci/devices/$GPU_SLOT.0/boot_vga" 2>/dev/null)" == "1" ]]; then
    echo "$GPU_SLOT is driving the host console; pick another card." >&2
    exit 1
  fi

  DEFAULT_USER=${SUDO_USER:-$(logname 2>/dev/null || id -un)}
  read -rp "Account to add to the libvirt group [$DEFAULT_USER]: " VM_USER
  VM_USER=${VM_USER:-$DEFAULT_USER}
  id "$VM_USER" >/dev/null || { echo "No such account: $VM_USER" >&2; exit 1; }

  read -rp "Install the Cockpit web console too? [y/N]: " WANT_COCKPIT
  read -rp "Reboot when the script finishes? [y/N]: " WANT_REBOOT

  SUDO="sudo"
  if [[ "$EUID" -eq 0 ]]; then
    SUDO=""
  elif ! sudo -v; then
    echo "This account cannot sudo. See the first-boot runbook." >&2
    exit 1
  fi

  # ---- Checks. Still nothing changed. ----

  echo "==> Checking the host can do this"

  CPU_VENDOR=$(awk -F': ' '/^vendor_id/ { print $2; exit }' /proc/cpuinfo)
  case "$CPU_VENDOR" in
    AuthenticAMD) IOMMU_ARGS="amd_iommu=on iommu=pt" ;;
    GenuineIntel) IOMMU_ARGS="intel_iommu=on iommu=pt" ;;
    *) echo "Unrecognised CPU vendor '$CPU_VENDOR'." >&2; exit 1 ;;
  esac

  grep -qw -e vmx -e svm /proc/cpuinfo ||
    { echo "No vmx/svm flag: virtualisation is off in firmware." >&2; exit 1; }

  [[ -f /etc/default/grub ]] ||
    { echo "No /etc/default/grub; this host does not boot with GRUB." >&2; exit 1; }
  [[ -f /etc/initramfs-tools/modules ]] ||
    { echo "No /etc/initramfs-tools/modules; this host does not use" >&2
      echo "initramfs-tools." >&2; exit 1; }

  mapfile -t GPU_IDS < <(lspci -n -s "$GPU_SLOT." | awk '{ print $3 }')
  mapfile -t GPU_MODULES < <(
    lspci -nnk -s "$GPU_SLOT." |
      sed -n 's/^[[:space:]]*Kernel modules:[[:space:]]*//p' |
      tr ',' '\n' | sed 's/[[:space:]]//g' | grep -v '^$' | sort -u
  )
  [[ ${#GPU_IDS[@]} -gt 0 ]] ||
    { echo "Found no PCI functions at $GPU_SLOT." >&2; exit 1; }
  GPU_ID_LIST=$(IFS=,; echo "${GPU_IDS[*]}")

  IOMMU_LIVE=0
  GROUP_DIR="/sys/bus/pci/devices/$GPU_SLOT.0/iommu_group/devices"
  if [[ -d "$GROUP_DIR" ]]; then
    IOMMU_LIVE=1
    mapfile -t STRANGERS < <(
      for dev in "$GROUP_DIR"/*; do
        base=${dev##*/}
        [[ "$base" == "$GPU_SLOT."* ]] || echo "$base"
      done
    )
    if [[ ${#STRANGERS[@]} -gt 0 ]]; then
      echo "$GPU_SLOT shares its IOMMU group with:" >&2
      printf '  %s\n' "${STRANGERS[@]}" >&2
      echo "VFIO hands over a group whole or not at all. Move the card to" >&2
      echo "another PCIe slot and run this again." >&2
      exit 1
    fi
  fi

  echo "    $CPU_VENDOR, ${#GPU_IDS[@]} function(s) at $GPU_SLOT: $GPU_ID_LIST"

  # ---- Everything below here changes the machine. ----

  echo "==> Installing the virtualisation stack"
  export DEBIAN_FRONTEND=noninteractive
  $SUDO apt-get update
  $SUDO apt-get install -y libvirt-daemon-system libvirt-clients virtinst \
      qemu-system-x86 ovmf
  if [[ "$WANT_COCKPIT" == [Yy]* ]]; then
    $SUDO apt-get install -y cockpit cockpit-machines
  fi

  echo "==> Adding $VM_USER to the libvirt group"
  $SUDO usermod -aG libvirt "$VM_USER"

  echo "==> Making libvirt's default network start at boot"
  if $SUDO virsh net-info default >/dev/null 2>&1; then
    $SUDO virsh net-autostart default
    if [[ "$($SUDO virsh net-info default | awk '/^Active:/ { print $2 }')" != "yes" ]]; then
      $SUDO virsh net-start default
    fi
  else
    echo "    no 'default' network is defined; skipping"
  fi

  echo "==> Reserving $GPU_SLOT for vfio-pci"
  {
    echo "# NVIDIA GPU at $GPU_SLOT, reserved for VFIO passthrough."
    echo "# Written by the Debian 13 GPU passthrough runbook."
    echo "options vfio-pci ids=$GPU_ID_LIST"
    for mod in "${GPU_MODULES[@]}"; do
      echo "softdep $mod pre: vfio-pci"
    done
  } | $SUDO tee /etc/modprobe.d/vfio.conf >/dev/null

  echo "==> Adding the VFIO modules to the initramfs"
  for mod in vfio vfio_iommu_type1 vfio_pci; do
    if ! grep -qxF "$mod" /etc/initramfs-tools/modules; then
      echo "$mod" | $SUDO tee -a /etc/initramfs-tools/modules >/dev/null
    fi
  done

  echo "==> Turning the IOMMU on at the kernel command line"
  $SUDO cp -n /etc/default/grub /etc/default/grub.dist
  CMDLINE=$( . /etc/default/grub; echo "${GRUB_CMDLINE_LINUX_DEFAULT:-}" )
  CMDLINE=$(printf '%s' "$CMDLINE" | tr ' ' '\n' |
            grep -vE '^(amd_iommu|intel_iommu|iommu)=' | paste -sd' ' - || true)
  NEW_CMDLINE="${CMDLINE:+$CMDLINE }$IOMMU_ARGS"
  if grep -q '^GRUB_CMDLINE_LINUX_DEFAULT=' /etc/default/grub; then
    $SUDO sed -i \
      "s|^GRUB_CMDLINE_LINUX_DEFAULT=.*|GRUB_CMDLINE_LINUX_DEFAULT=\"$NEW_CMDLINE\"|" \
      /etc/default/grub
  else
    echo "GRUB_CMDLINE_LINUX_DEFAULT=\"$NEW_CMDLINE\"" |
      $SUDO tee -a /etc/default/grub >/dev/null
  fi

  echo "==> Rebuilding the initramfs and the boot configuration"
  $SUDO update-initramfs -u -k all
  $SUDO update-grub

  echo
  echo "Done, but not in effect: the running kernel still has $GPU_SLOT on"
  echo "its host driver. The card moves at the next boot."
  echo "  vfio-pci will claim ids=$GPU_ID_LIST"
  echo "  kernel command line: $NEW_CMDLINE"
  echo "  $VM_USER is in the libvirt group - log out and back in for it."
  if [[ "$IOMMU_LIVE" -eq 0 ]]; then
    echo "  The IOMMU is not active yet, so the card's isolation could not"
    echo "  be checked. Check it after the reboot, before building a VM."
  fi
  if [[ "$WANT_COCKPIT" == [Yy]* ]]; then
    echo "  Cockpit is on https://$(hostname -f 2>/dev/null || hostname):9090/"
  fi
  echo
  echo "After the reboot, check:"
  echo "  lspci -nnk -s $GPU_SLOT.   # every function must say vfio-pci"
  echo "  ls /sys/class/iommu/        # must not be empty"
  echo "  ls -l /dev/kvm"

  if [[ "$WANT_REBOOT" == [Yy]* ]]; then
    echo
    echo "Rebooting in 5 seconds. Ctrl-C to stay up."
    sleep 5
    $SUDO systemctl reboot
  fi
)

How it works

Choosing the card

lspci -D -d '10de::0300' and -d '10de::0302' list PCI devices whose vendor is NVIDIA — 10de — and whose class is VGA compatible controller or 3D controller. Compute cards present as the latter and have no VGA class at all, which is why both are asked for. -D prints the PCI domain, so the slots come out as 0000:01:00 and every later sysfs path can be built from them without guessing.

boot_vga in sysfs is the kernel's record of which adapter the firmware booted with, and it is the one check worth making before anything else. A card that is driving the host console cannot also be given to a guest: the host loses its display, and the card's firmware has already been run by the host in a way the guest's reset expects not to find. The script will not proceed with such a card, and if every NVIDIA card in the box is that card, it stops and sends you to the firmware setup.

The slot you type is normalised to include a domain if you left it out, so 01:00 and 0000:01:00 both work.

Deciding the kernel arguments

vendor_id from /proc/cpuinfo picks between amd_iommu=on and intel_iommu=on. iommu=pt — passthrough mode — goes on either way: it leaves devices the host still owns on the untranslated DMA path, which is faster, while the devices actually assigned to VFIO still get full translation.

The vmx or svm flag is what the CPU reports when hardware virtualisation is switched on in firmware. It is checked here rather than looking at /dev/kvm, because that device does not appear until the KVM modules load, which has not happened yet on a host where QEMU was never installed.

Reading the card's own configuration off the card

lspci -n -s "$GPU_SLOT." — note the trailing dot, which selects every function of the slot rather than function zero — prints the vendor:device pair for each. A graphics card is at least two functions, the display controller and an HDMI/DisplayPort audio controller, and both have to go to the guest. Reading them from the card is why nothing in this script needs editing for your hardware.

Kernel modules: from lspci -k names every driver that could claim each function, and each one becomes a softdep. That list is read off the running host rather than hard-coded, so a host where the proprietary nvidia driver is installed gets softdep nvidia pre: vfio-pci without anyone remembering to add it.

Checking the card is on its own

An IOMMU group is the smallest set of devices that can be isolated from each other, and VFIO hands over a group whole or not at all. If an NVMe drive or a USB controller shares the group, it would have to go to the guest too. The script lists any such device and stops, rather than reaching for an ACS override kernel patch, which defeats the isolation the group exists to provide — move the card to a different PCIe slot instead.

This check needs a live IOMMU, which is exactly what the script is in the middle of enabling. On a host where the IOMMU is not yet up, sysfs has no iommu_group link, the check is skipped, and the closing summary says so. Run the grouping check by hand after the reboot before you build a VM.

Installing the stack

cockpit-machines pulls in libvirt-clients, libvirt-dbus, and virtinst on its own, but they are named explicitly because the Cockpit half is optional and the stack has to work without it. qemu-system-x86 is the emulator and ovmf the UEFI firmware images — a guest that needs UEFI, which a passed-through GPU effectively does, cannot be created without the latter.

Debian 13 still ships the monolithic libvirtd, socket-activated, rather than the modular per-hypervisor daemons; virsh starting it on demand is why the network commands work immediately after install with nothing enabled by hand.

usermod -aG libvirt "$VM_USER" adds one supplementary group — the -a is what keeps it from replacing every supplementary group the account has. Cockpit and virsh both talk to libvirt as you, not as root, so without this the Virtual machines page loads read-only. Group membership is fixed when a session is created, so the account has to log out and back in.

virsh net-autostart default is what survives a reboot; net-start is only run when the network is not already up, because starting a running network is an error. Without that network, new VMs have no interface to attach to and the create dialog offers nothing usable.

Reserving the card

options vfio-pci ids=… tells vfio-pci which devices to claim when it loads. On its own that is not enough, because whichever driver reaches a device first keeps it: snd_hda_intel loads early for the host's onboard audio and takes the GPU's audio function on the way past, leaving the display controller on vfio-pci and the audio function on snd_hda_intel. libvirt then tries to wrest that function back when the VM starts and fails, because the host has the device open — so the passthrough works until the day it does not. The softdep … pre: vfio-pci lines are what prevent that, and they are the part people leave out.

The three modules appended to /etc/initramfs-tools/modules are there because a softdep can only prefer a module that exists in the same context, so VFIO has to be in the initramfs alongside the drivers it is racing. They are appended only when absent, so pasting this twice does not accumulate duplicates. Older guides also list vfio_virqfd; that module was folded into the VFIO core in Linux 6.2 and does not exist on Debian 13's 6.12 kernel.

Editing the kernel command line

/etc/default/grub is sourced rather than pattern-matched, so the value read is the one GRUB itself would use — the last assignment wins, and a commented-out line above it is ignored, both of which a sed would get wrong. Any existing amd_iommu=, intel_iommu=, or iommu= argument is dropped before the new pair is appended, which is what makes a second run idempotent instead of leaving the command line with the argument twice.

update-initramfs -u -k all rebuilds every installed kernel's initramfs, not just the running one, so the card still lands on vfio-pci if you boot an older kernel from the GRUB menu. update-grub writes the new command line into /boot/grub/grub.cfg.

Neither takes effect until the machine reboots — until then the running kernel still has the card on nouveau or nvidia, and a VM started now would fail to claim it. To undo all of it, delete /etc/modprobe.d/vfio.conf, restore /etc/default/grub from the grub.dist copy the script takes on its first run, remove the three vfio lines from /etc/initramfs-tools/modules, and run the same two rebuild commands.

After the reboot

lspci -nnk -s <slot>. must report vfio-pci on every function of the card. If one still shows its ordinary driver, the softdep for that driver is missing or misspelled — the name has to match the Kernel modules: line exactly, underscores and all. An empty /sys/class/iommu/ means the IOMMU is still off in firmware, whatever the kernel command line says, and nothing below that will work.