How to configure IOMMU device for nested Proxmox hypervisor (PVE) VM - PCIe Passthrough

Overview

Configuring PCIe Passthrough for Nested Virtualization on Proxmox

Summary:

If you are running bare-metal L0 (level 0) Proxmox (PVE) hypervisor with nested PVE hypervisor in L1 VM, and you wish to passthrough a PCIe device, physically attached to the bare-metal machine, to a VM inside the L1 VM (nested PVE), then this may be for you.

This was tested on a machine using Intel Xeon X5670, which has the required Virtualization Technology for Directed I/O (VT-d) support with Proxmox PVE v7.0-2.

Challenges:

I have a PCIe-to-SATA expansion card added to my computer and a SATA DVD drive was connected to it. Ultimately, I wanted the L2 VM to be able to fully control the drive.

Most PCIe passthrough guides found all over the Internet do not cover how one can set up passthrough for L2 VM residing in L1 nested hypervisor on top of L0 bare-metal hypervisor due to nested VM being two levels deep, which is needless for most people due to potential performance issues and no live migration support.

If you are like me, who prefers to experiment with passthrough on VMs directly on L1 PVEs due to clustering, then you have come to the right place.

Unfortunately, even Proxmox's PCIe passthrough guide doesn't cover nested virtualization. The minimal OS configuration needed for PCIe passthrough for supported Intel machine is updating the kernel command-line parameter, intel_iommu=on, which will be covered in later steps.

Requirements:

This guide assumes some familiarity with Proxmox (e.g. creating VMs).

There are two sets of requirements we need to fulfill as one is intended for nested virtualization and the other is for PCIe passthrough.

The below requirements are intended for the L1 PVE VM. Not the L2 VM guest.

Nested Virtualization:
  • CPU on bare-metal machine requires hardware-assisted virtualization extensions (e.g. Intel VT-x, AMD-V) which needs to be enabled in BIOS/UEFI.
  • Nested PVE (L1) using the same PVE version as the bare-metal PVE (L0)
  • Nested PVE VM needs to have CPU set to host type (ex. qm set --cpu host )
  • Nested PVE VM needs to use ovmf BIOS implementation (ex. qm set --bios ovmf )
  • Nested PVE VM requires q35 QEMU machine type (ex. qm set --machine q35 )

NOTE: It's extremely important to use q35 QEMU machine type and ovmf BIOS implementation together before booting up the machine. Otherwise, a kernel panic may occur, even at the bare-metal level and a hard reboot may be required to unseize the machine.

Nested virtualization may already be enabled for you if you are running the latest version of PVE. PVE 7.0 seems to have it enabled by default. To check, run cat /sys/module/kvm_intel/parameters/nested on L0 and you'd get either Y or N.

You can also configure the options through the web GUI instead of qm command. It's recommended to add only EFI disk on a storage location with no extra options but not required.

Screenshot of Create Virtual Machine - System option with OVMF BIOS and q35 machine selected

PCIe Passthrough:
  • Bare-metal requires CPU and motherboard with IOMMU (I/O Memory Management Unit) interrupt remapping support (e.g. Intel VT-d, AMD-Vi). Many modern CPUs and motherboards support this. This needs to be enabled in BIOS/UEFI settings.
  • For nested PVE, CPU also needs to be host type, ovmf BIOS implementation is required, and q35 QEMU machine type is necessary. This requirement is already outlined in the Nested Virtualization section.

Configuring Passthrough:

Enable IOMMU for L0:

Before we can proceed, we'd need to make sure the L0 hypervisor (bare-metal PVE) has IOMMU enabled, which can be done by tuning the kernel command-line parameter.

As mentioned in Proxmox documentation, you need to add intel_iommu=on in the bootloader configuration file depending on which one one you are using. Usually BIOS are found on legacy systems whereas UEFI are on modern systems. GRUB bootloader is typically for BIOS system, and EFI bootloader is for UEFI. Please see Proxmox documentation on editing the kernel command-line.

A reboot would be required after updating the kernel parameter in bootloader configuration file.

To verify, run dmesg | grep -e DMAR -e IOMMU -e AMD-Vi and if you see lines containing IOMMU, Directed I/O, or Interrupt Remapping being enabled, then you are set.

Enable IOMMU for L1 PVE:

This assumes L1 PVE has been created and you are ready to create a VM from L1 PVE virtual machine.

Before we can get to the next step, you also need to modify the kernel parameter for the L1 PVE the same way you did for L0 PVE in the previous section.

Create L2 VM:

You should already have the ability to add PCI device to the L1 VM through the Hardware settings.

At this point, you should be able to passthrough a PCIe device from L0 to L1 PVE. It would be listed in lspci -nnk output in L1 PVE's shell.

Proceed to create the L2 VM using any compatible Linux guest OS of your choice (Windows hasn't been tested) with the following virtual hardware requirements.

  • OVMF BIOS implementation must be used
  • q35 machine type is required

Go ahead attempt to add the passthrough PCI device as well.

What did you get? A warning like the one below?

No IOMMU detected error while adding PCI device to L2 VM

Despite updating the kernel parameter, you are faced with this error until you add a vIOMMU device to the L1 PVE, which was missing all along.

Where is the vIOMMU option?

You may wonder where you may have missed this option while creating the PVE. It turns out, it's not natively included with Proxmox tools at the time of this writing as VT-d emulation would need to be added in QEMU.

How VM is run on PVE is through the /usr/bin/qemu-system-x86_64 command. You would see that /usr/bin/kvm is a symbolic link to the previous command that powers the VMs on PVE.

Run the qm showcmd command for any configured VM of your choice, and you would see a very long kvm command. The long command derives from the configurations that you'd make to the VM.

To view the VM configuration via command, run qm config , which is human-readable as all options are in alphabetical order. Please see more information on qm command here.

According to the QEMU wiki article previously mentioned, the options needed to add the vIOMMU device to L1 PVE can theoretically be done through the qm set --args command.

qm set 100 --args '-device intel-iommu,intremap=on,caching-mode=on -machine accel=kvm,kernel-irqchip=split'

The above change would also reflect in the qm config and qm showcmd commands.

However... Because qm set --args would simply append the missing options to the end of the long kvm command, it would not work. The QEMU article said the intel-iommu device must be specificed before all other device parameters that you'd see in the kvm command.

You would have to shutdown the PVE, modifying the kvm command and add the missing options. The major drawback is that this change would have to be made each time you boot if you need IOMMU support for passthrough.

Adding vIOMMU to L1 PVE:

Fortunately, I've created a Python script to simplify the job. Hopefully, one day the Proxmox team can embed the vIOMMU option in a future release.

Just simply go to the L0 PVE's shell and run the commands below using the L1 PVE's VM ID. The script will not send a reboot signal and it'd passively wait for the VM to shutdown (not reboot as the script would be booting back up the VM).

wget https://raw.githubusercontent.com/bashtheshell/IOMMU-nested-pve/main/add_vIOMMU.py
python3 add_vIOMMU.py 
   

   

Please note that the above script may not be compatible with older version of PVE as this was tested on 7.0. At least Python 3.5 would be required.

Also, the script would have to be run at each boot when IOMMU is needed.

Re-attempt to add passthrough PCI device to L2 VM:

Previously, you may have received an error message on L2 VM that IOMMU isn't detected. Now that should no longer be the case.

Upon booting up the L2 VM, you should be able to see it in lspci -nnk output inside the guest OS. If yes, YOU ARE ALL SET!

If you are still facing challenges, then chances are you may be required to do further tweaking. Please review Proxmox's PCIe passthrough documentations.

Special Thanks:

Thanks to thenickdude for his contribution to my Reddit post, which steered me to the solution.

And of course, the Proxmox and QEMU/KVM teams for making all things possible for us homelabbers and tinkers.

Owner
Travis Johnson
Linux system administration enthusiast. Loves to tinker with programming and networking.
Travis Johnson
iot-dashboard: Fully integrated architecture platform with a dashboard for Logistics Monitoring, Internet of Things.

Fully integrated architecture platform with a dashboard for Logistics Monitoring, Internet of Things. Written in Python. Flask applicati

2 Jul 29, 2022
Controlling fireworks with micropython

Controlling-fireworks-with-micropython How the code works line 1-4 from machine

Montso Mokake 1 Jan 08, 2022
Volkswagen ID component for Home Assistant

Volkswagen ID component for Home Assistant This folder contains both a generic Python 3 library for the Volkswagen ID API and a component for Home Ass

55 Jan 07, 2023
Skykettle ha - Redmond SkyKettle integration for Home Assistant

Redmond SkyKettle integration for Home Assistant This integration allows to cont

Alexey 'Cluster' Avdyukhin 48 Jan 06, 2023
Custom component for interacting with Octopus Energy

Home Assistant Octopus Energy ** WARNING: This component is currently a work in progress ** Custom component built from the ground up to bring your Oc

David Kendall 116 Jan 02, 2023
Modi2-firmware-updater - MODI+ Firmware Updater With Python

MODI+ Firmware Updater 실행 준비 python3(파이썬3.9 혹은 그 이상의 버전)를 컴퓨터에 설치 python3 -m pip

LUXROBO 1 Feb 04, 2022
Plug and Play on Internet of Things with LoRa wireless modulation.

IoT-PnP Plug and Play on Internet of Things with LoRa wireless modulation. Device Side In the '505_PnP' folder has a modified ardunino template code s

Lambert Yang 1 May 19, 2022
Python script: Enphase Envoy mqtt json for Home Assistant

A Python script that takes a real time stream from Enphase Envoy and publishes to a mqtt broker. This can then be used within Home Assistant or for other applications. The data updates at least once

29 Dec 27, 2022
HomeAssistant - Polyaire AirTouch 4 Integration

HomeAssistant - Polyaire AirTouch 4 Integration Custom integration to add an AirTouch 4 AC Controller Installation: Copy contents of custom_components

7 Aug 05, 2022
This application works with serial communication. Use a simple gui to send and receive serial data from arduino and control leds and motor direction

This application works with serial communication. Use a simple gui to send and receive serial data from arduino and control leds and motor direction

ThyagoKZKR 2 Jul 18, 2022
A small Python app to converse between MQTT messages and 433MHz RF signals.

mqtt-rf-bridge A small Python app to converse between MQTT messages and 433MHz RF signals. This acts as a bridge between Paho MQTT and rpi-rf. Require

David Swarbrick 3 Jan 27, 2022
Jarvis: a personal assistant which can help you to manage your system

Jarvis Jarvis is personal AI based assistant which can help you to manage stuff in your computer. This is demo but I decided to make it more better so

2 Jun 02, 2022
Python library for the Phomemo m02s bluetooth thermal printer

Phomemo M02S Python library This is a basic Python library for controlling the Phomemo M02S bluetooth thermal printer. It probably only works on Mac &

Stargirl Flowers 28 Nov 07, 2022
智能无人机路径规划仿真系统是一个具有操作控制精细、平台整合性强、全方向模型建立与应用自动化特点的软件

Drone智能无人机路径规划仿真系统是一个具有操作控制精细、平台整合性强、全方向模型建立与应用自动化特点的软件。它以A、B两国在C区开展无人机战争为背景,该系统的核心功能是通过仿真平台规划无人机航线,并进行验证输出,数据可导入真实无人机,使其按照规定路线精准抵达战场任一位置,支持多人多设备编队联合行动。

wwy 349 Jan 03, 2023
Sleep As Android integration for Home Assistant

Sleep As Android custom integration This integration will allow you to get events from your SleepAsAndroid application in a form of the sensor states

Igor 84 Dec 30, 2022
Software framework to enable agile robotic assembly applications.

ConnTact Software framework to enable agile robotic assembly applications. (Connect + Tactile) Overview Installation Development of framework was done

Southwest Research Institute Robotics 29 Dec 01, 2022
3D-printable hexagonal mirror array capable of reflecting sunlight into arbitrary patterns

3D-printable hexagonal mirror array capable of reflecting sunlight into arbitrary patterns

Ben Bartlett 2.3k Dec 30, 2022
circuitpython version of PyBasic for microcontrollers

cPyBasic Circuitpython version of PyBasic for microcontrollers Current version work only for Adafruit titano & CardKB for now. The origninal PyBasic w

BeBoXoS 3 Nov 14, 2021
Segger Embedded Studio project for building & debugging Flipper Zero firmware.

Segger Embedded Studio project for Flipper Zero firmware Установка Добавить данный репозиторий в качестве сабмодуля в корень локальной копии репозитор

25 Dec 28, 2022
a weather application for the raspberry pi and the Pimorioni Inky pHAT.

raspi-weather a weather application for the raspberry pi and the Inky pHAT

Derek Caelin 59 Oct 24, 2022