How to configure IOMMU device for nested Proxmox hypervisor (PVE) VM - PCIe Passthrough

Overview

Configuring PCIe Passthrough for Nested Virtualization on Proxmox

Summary:

If you are running bare-metal L0 (level 0) Proxmox (PVE) hypervisor with nested PVE hypervisor in L1 VM, and you wish to passthrough a PCIe device, physically attached to the bare-metal machine, to a VM inside the L1 VM (nested PVE), then this may be for you.

This was tested on a machine using Intel Xeon X5670, which has the required Virtualization Technology for Directed I/O (VT-d) support with Proxmox PVE v7.0-2.

Challenges:

I have a PCIe-to-SATA expansion card added to my computer and a SATA DVD drive was connected to it. Ultimately, I wanted the L2 VM to be able to fully control the drive.

Most PCIe passthrough guides found all over the Internet do not cover how one can set up passthrough for L2 VM residing in L1 nested hypervisor on top of L0 bare-metal hypervisor due to nested VM being two levels deep, which is needless for most people due to potential performance issues and no live migration support.

If you are like me, who prefers to experiment with passthrough on VMs directly on L1 PVEs due to clustering, then you have come to the right place.

Unfortunately, even Proxmox's PCIe passthrough guide doesn't cover nested virtualization. The minimal OS configuration needed for PCIe passthrough for supported Intel machine is updating the kernel command-line parameter, intel_iommu=on, which will be covered in later steps.

Requirements:

This guide assumes some familiarity with Proxmox (e.g. creating VMs).

There are two sets of requirements we need to fulfill as one is intended for nested virtualization and the other is for PCIe passthrough.

The below requirements are intended for the L1 PVE VM. Not the L2 VM guest.

Nested Virtualization:
  • CPU on bare-metal machine requires hardware-assisted virtualization extensions (e.g. Intel VT-x, AMD-V) which needs to be enabled in BIOS/UEFI.
  • Nested PVE (L1) using the same PVE version as the bare-metal PVE (L0)
  • Nested PVE VM needs to have CPU set to host type (ex. qm set --cpu host )
  • Nested PVE VM needs to use ovmf BIOS implementation (ex. qm set --bios ovmf )
  • Nested PVE VM requires q35 QEMU machine type (ex. qm set --machine q35 )

NOTE: It's extremely important to use q35 QEMU machine type and ovmf BIOS implementation together before booting up the machine. Otherwise, a kernel panic may occur, even at the bare-metal level and a hard reboot may be required to unseize the machine.

Nested virtualization may already be enabled for you if you are running the latest version of PVE. PVE 7.0 seems to have it enabled by default. To check, run cat /sys/module/kvm_intel/parameters/nested on L0 and you'd get either Y or N.

You can also configure the options through the web GUI instead of qm command. It's recommended to add only EFI disk on a storage location with no extra options but not required.

Screenshot of Create Virtual Machine - System option with OVMF BIOS and q35 machine selected

PCIe Passthrough:
  • Bare-metal requires CPU and motherboard with IOMMU (I/O Memory Management Unit) interrupt remapping support (e.g. Intel VT-d, AMD-Vi). Many modern CPUs and motherboards support this. This needs to be enabled in BIOS/UEFI settings.
  • For nested PVE, CPU also needs to be host type, ovmf BIOS implementation is required, and q35 QEMU machine type is necessary. This requirement is already outlined in the Nested Virtualization section.

Configuring Passthrough:

Enable IOMMU for L0:

Before we can proceed, we'd need to make sure the L0 hypervisor (bare-metal PVE) has IOMMU enabled, which can be done by tuning the kernel command-line parameter.

As mentioned in Proxmox documentation, you need to add intel_iommu=on in the bootloader configuration file depending on which one one you are using. Usually BIOS are found on legacy systems whereas UEFI are on modern systems. GRUB bootloader is typically for BIOS system, and EFI bootloader is for UEFI. Please see Proxmox documentation on editing the kernel command-line.

A reboot would be required after updating the kernel parameter in bootloader configuration file.

To verify, run dmesg | grep -e DMAR -e IOMMU -e AMD-Vi and if you see lines containing IOMMU, Directed I/O, or Interrupt Remapping being enabled, then you are set.

Enable IOMMU for L1 PVE:

This assumes L1 PVE has been created and you are ready to create a VM from L1 PVE virtual machine.

Before we can get to the next step, you also need to modify the kernel parameter for the L1 PVE the same way you did for L0 PVE in the previous section.

Create L2 VM:

You should already have the ability to add PCI device to the L1 VM through the Hardware settings.

At this point, you should be able to passthrough a PCIe device from L0 to L1 PVE. It would be listed in lspci -nnk output in L1 PVE's shell.

Proceed to create the L2 VM using any compatible Linux guest OS of your choice (Windows hasn't been tested) with the following virtual hardware requirements.

  • OVMF BIOS implementation must be used
  • q35 machine type is required

Go ahead attempt to add the passthrough PCI device as well.

What did you get? A warning like the one below?

No IOMMU detected error while adding PCI device to L2 VM

Despite updating the kernel parameter, you are faced with this error until you add a vIOMMU device to the L1 PVE, which was missing all along.

Where is the vIOMMU option?

You may wonder where you may have missed this option while creating the PVE. It turns out, it's not natively included with Proxmox tools at the time of this writing as VT-d emulation would need to be added in QEMU.

How VM is run on PVE is through the /usr/bin/qemu-system-x86_64 command. You would see that /usr/bin/kvm is a symbolic link to the previous command that powers the VMs on PVE.

Run the qm showcmd command for any configured VM of your choice, and you would see a very long kvm command. The long command derives from the configurations that you'd make to the VM.

To view the VM configuration via command, run qm config , which is human-readable as all options are in alphabetical order. Please see more information on qm command here.

According to the QEMU wiki article previously mentioned, the options needed to add the vIOMMU device to L1 PVE can theoretically be done through the qm set --args command.

qm set 100 --args '-device intel-iommu,intremap=on,caching-mode=on -machine accel=kvm,kernel-irqchip=split'

The above change would also reflect in the qm config and qm showcmd commands.

However... Because qm set --args would simply append the missing options to the end of the long kvm command, it would not work. The QEMU article said the intel-iommu device must be specificed before all other device parameters that you'd see in the kvm command.

You would have to shutdown the PVE, modifying the kvm command and add the missing options. The major drawback is that this change would have to be made each time you boot if you need IOMMU support for passthrough.

Adding vIOMMU to L1 PVE:

Fortunately, I've created a Python script to simplify the job. Hopefully, one day the Proxmox team can embed the vIOMMU option in a future release.

Just simply go to the L0 PVE's shell and run the commands below using the L1 PVE's VM ID. The script will not send a reboot signal and it'd passively wait for the VM to shutdown (not reboot as the script would be booting back up the VM).

wget https://raw.githubusercontent.com/bashtheshell/IOMMU-nested-pve/main/add_vIOMMU.py
python3 add_vIOMMU.py 
   

   

Please note that the above script may not be compatible with older version of PVE as this was tested on 7.0. At least Python 3.5 would be required.

Also, the script would have to be run at each boot when IOMMU is needed.

Re-attempt to add passthrough PCI device to L2 VM:

Previously, you may have received an error message on L2 VM that IOMMU isn't detected. Now that should no longer be the case.

Upon booting up the L2 VM, you should be able to see it in lspci -nnk output inside the guest OS. If yes, YOU ARE ALL SET!

If you are still facing challenges, then chances are you may be required to do further tweaking. Please review Proxmox's PCIe passthrough documentations.

Special Thanks:

Thanks to thenickdude for his contribution to my Reddit post, which steered me to the solution.

And of course, the Proxmox and QEMU/KVM teams for making all things possible for us homelabbers and tinkers.

Owner
Travis Johnson
Linux system administration enthusiast. Loves to tinker with programming and networking.
Travis Johnson
Raspberry Pi Pico development platform for PlatformIO

Raspberry Pi Pico development platform for PlatformIO A few words in the beginning Before experimental please Reinstall the platform Version: 1.0.0 Th

Georgi Angelov 160 Dec 23, 2022
E-Ink Magic Calendar that automatically syncs to Google Calendar and runs off a battery powered Raspberry Pi Zero

E-Ink Magic Calendar that automatically syncs to Google Calendar and runs off a battery powered Raspberry Pi Zero

2.8k Dec 30, 2022
Tool to create 3D printable terrain with integrated path/road part files (Single material 3d printer)

BACKGROUND This has been an ongoing project of mine for a few months now. I run trails a lot and original the goal was to create a function to combine

9 Apr 26, 2022
Count the number of people around you 👨‍👨‍👦 by monitoring wifi signals 📡 .

howmanypeoplearearound Count the number of people around you 👨‍👨‍👦 by monitoring wifi signals 📡 . howmanypeoplearearound calculates the number of

Zack 6.7k Jan 07, 2023
A simple portable USB MIDI controller based on Raspberry-PI Pico and a 16-button keypad, written in Circuit Python

RPI-Pico-16-BTn-MIDI-Controller-using-CircuitPython A simple portable USB MIDI controller based on Raspberry-PI Pico, written in Circuit Python. Link

Rounak Dutta 3 Dec 04, 2022
It is a serial communicator(controller, receiver...), communicate with sensor LP20 which is a laser ranger.

Intro It is a serial communicator(controller, receiver...), communicate with sensor LP20 which is a laser ranger. Its datasheet is contained in this r

3 Sep 19, 2022
Ha-rpi gpio - Home Assistant Raspberry Pi GPIO Integration

Home Assistant Raspberry Pi GPIO custom integration This is a spin-off from the

Shay Levy 98 Dec 24, 2022
A ESP32 project template with a web interface built in React

ESP AP Webserver demo.mp4 This is my experiment with "mobile app development" for the ESP32. The project consists of two parts, the ESP32 code and the

8 Dec 15, 2022
Inykcal is a software written in python for selected E-Paper displays.

Inykcal is a software written in python for selected E-Paper displays. It converts these displays into useful information dashboards. It's open-source, free for personal use, fully modular and user-f

Ace 727 Jan 02, 2023
Fener ROS2 package version 2

Fener's ROS2 codes that runs on the vehicle. This node contains basic sensing and actuation nodes for vehicle control. Also example applications will be added.

Muhammed Sezer 1 Jan 18, 2022
Yet another automation project because a smart light is more than just on or off.

Automate home Yet another home automation project because a smart light is more than just on or off. Overview When talking about home automation there

Maja Massarini 62 Oct 10, 2022
A circle of LEDs

This repository contains all the design files, production files and example code for a simple circular LED display.

Pim de Groot 15 Aug 21, 2022
Интеграция Home Assistant с ЛК "Интер РАО"

ЕЛК ЖКХ «Интер РАО» для Home Assistant Предоставление информации о текущем состоянии ваших аккаунтов в ЕЛК ЖКХ. Введение @ TODO @ Установка Посредство

Alexander Ryazanov 27 Nov 05, 2022
Tools and documentation to aid in modifying the ADI ADALM Pluto firmware

Pluto firmware modifications This repository contains tools and documentation to aid in modifying the ADI ADALM Pluto firmware. Extraction of the Plut

Daniel Estévez 28 Dec 21, 2022
Modi2-firmware-updater - MODI+ Firmware Updater With Python

MODI+ Firmware Updater 실행 준비 python3(파이썬3.9 혹은 그 이상의 버전)를 컴퓨터에 설치 python3 -m pip

LUXROBO 1 Feb 04, 2022
Imbalaced Classification and Robust Semantic Segmentation

Imbalaced Classification and Robust Semantic Segmentation This repo implements two algoritms. The imbalance clibration (IC) algorithm for image classi

24 Jul 23, 2022
🌱 - WebhookHard◞ Fines Educativos ◟

v1.0.0 WebhookHardware ¿Que es WebhookHardware? WebhookHardware se trata de un proyecto tratado para sacar informacion sobre el hardware de tus victim

3 Jun 14, 2021
DIY split-flap display

The goal is to make a low-cost display that's easy to fabricate at home in small/single quantities (e.g. custom materials can be ordered from Ponoko or similar, and other hardware is generally availa

Scott Bezek 2.5k Jan 05, 2023
A script for performing OTA update over BLE on ESP32

A script for performing OTA update over BLE on ESP32

Felix Biego 18 Dec 15, 2022
Robo Arm :: Rigging is a rigging addon for Blender that helps animating industrial robotic arms.

Robo Arm :: Rigging Robo Arm :: Rigging is a rigging addon for Blender that helps animating industrial robotic arms. It construct serial links(a kind

2 Nov 18, 2021