A couple months ago I built a new AMD Zen 2 computer. There was nothing wrong with my previous Ryzen 1700 workstation, or the Hades Canyon NUC that I was also swapping off between, but those that have being paying some attention to PC hardware might understand the excitement (Zen 2 marks a milestone, cementing not just AMD’s price/performance and multi-threaded performance lead over Intel, but also matching IPC and introducing the first chiplet-based design to the market). It doesn’t hurt that I’ve done very well on AMD stock the past few years, so I sort of feel justified with some more frequent upgrades.

For the past few years I’ve been running Linux as my primary computing environment, but have had to dual boot into Windows for most of my VR or gaming. One of my goals for this new system was to see if I could do this in a virtual machine and avoid the inconvenient process of rebooting with my new system.

Luckily, due to the interest in the past few years, driven both by enthusiasts and the demands of cloud gaming, virtualization with hardware pass-through has gone from black magic to merely bleeding edge. This is generally referred to as VFIO, referring to how devices are passed through to the virtual machine.

This is a summary of what I needed to do to get a pretty fully working setup (having decades of experience with Linux, but none with VFIO, KVM, QEMU, etc) in August 2019. There are plenty of sharp edges and caveats still, so those looking for support should also check out r/VFIO and the L1T VFIO forums.

VFIO is very hardware dependent (specifically for IOMMU groups).

Hardware

• ASUS ROG Crosshair VIII Hero Wi-Fi X570 (0702 and 0803 BIOS)
• AMD 3700X @ stock (crashes w/ PBO, FCLK 1800; will wait for updates to tweak)
• 4 x 16GB Crucial Ballistix Sport LT 3200@CL16 @ 3200 14-18-18-18-36
• Sabrent 2TB Rocket NVMe 4.0 M.2 (Linux Host)
• Samsung 1TB 960 EVO M.2 (Win10 Guest)
• Sapphire Nitro+ RX 470 4GB (on PCIEX16_1 for Linux Host)
• EVGA GeForce GTX 1080 Ti SC2 (on PCIEX16_2 for Win10 Guest)
• Inateck KT4007 PCI-E USB 3.0 Controller w/ Renasas uPD720201 (on PCIEX16_3 for Win10 Guest)

Software

• Arch Linux (5.2.5-arch1-1-ARCH) Host – no problems w/ kernel updates
• Windows 10 Guest (build 1903)
• qemu-4.0.0-3, libvirt 5.5.0-1, virt-manager 2.2.0-2, edk2-ovmf 20180815-1, ebtables 2.0.10_4-7, dnsmasq 2.80-4, bridge-utils 1.6-3

Motherboard notes

• The ASUS BIOS currently really sucks (Note: haven’t gotten around to the beta ABBA update), it seems to die in lots of ways it shouldn’t, including boot looping when PBO is enabled, and requiring CSM mode to boot with both an RX 470 and RX 570 I tried (I set everything network to Ignore and everything else to UEFI only as bootup is slowed significantly by this.)
• I was getting hard shutdowns in mprime – I ended up finding a tip that DRAM Current in the DigiVRM section of the BIOS needs to be set to 130% to prevent this.
• IOMMU groupings is decent (30 groups, every PCIe slot I’ve tried in its own group) however all the USB controllers are “[AMD] Matisse USB 3.0 Host Controller” and in board groups, see USB section for more details.

Windows Setup

I set up the Windows drive and GPU first by itself (including AMD Chipset drivers, SteamVR, some initial tuning/testing tools) and then shifted the M.2 over and setup Linux on the primary M.2.

Dual boot was as simple as copying \EFI\Microsoft\Boot\bootmgfw.efi over to my systemd-boot EFI partition on my new main M.2 once I did the switch.

IOMMU and VFIO

The primary guide I used for setup was Arch Wiki’s PCI passthrough via OVMF. Things I needed to do:

• Enable SVM in AMD CPU Features in the BIOS
• Add kernel parameters amd_iommu=on iommu=pt
• Check dmesg for IOMMU groups and use the bash script to check groups (see the above-referenced guide), which worked without issue
• Added the GPU and USB card PCI IDs to /etc/modeprobe.d/vfio.conf and the proper modules to /etc/mkinitcpio.conf and run mkinitcpio -p linux (or linux-vfio if you want to manually test first)

Besides discovering the CSM issue w/ my Polaris cards, this part was all pretty straightforward.

OVMF-based Guest VM

This took a bit more fiddling than I would have liked. I had to install a few more packages than was listed:

qemu libvirt ovmf virt-manager ebtables dnsmasq bridge-utils openbsd-netcat

I also ran into issues with the OVMF UEFI loading (you are supposed to be able to specify it in /etc/libvirt/qemu.conf by adding it to the nvram var like:

nvram = [
	"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"
]

But this didn’t work (you should see it loading a Tiano graphic vs SeaBIOS if it is) and I had to learn and fiddle with virsh -c qemu:///system until I could get it right. I ended clearing the nvram setting, using the edk2-ovmf package’s firmware and manually updated my XML (note virsh will autocorrect/parse things on save so if it’s eating settings you need to change things up):

 <os>
    <type arch='x86_64' machine='pc-q35-4.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/guest_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>

With this I was able to create a q35 VM that uses an existing storage device (pointing to the raw device as SATA – I couldn’t get scsi or virtio working, but running CrystalMark gave me 2GB read/writes so I didn’t try too hard to get it booting with the other methods after that). Here’s the XML config, since that part of the setup in the GUI was pretty confusing IMO (I ended up using virsh to edit a lot of XML directly vs the virtmanager GUI):

<disk type='block' device='disk'>
    <driver name='qemu' type='raw' cache='none' io='native'/>
    <source dev='/dev/disk/by-id/nvme-YOUR-DISK-HERE'/>
    <target dev='sda' bus='sata'/>
    <address type='drive' controller='0' bus='0' target='0' unit='0'/>
 </disk>

I added the GPU/HDMI device, USB board and also the Realtek 2.5Gb (which I didn’t use VFIO for since it doesn’t have drivers by default in the default Arch kernel anyway) as devices to the vm. I’ve actually disabled the bridged network so that Windows uses the Realtek device as the bridging seems to put a bit of extra load on my system.

I use the Nvidia workaround so the drivers don’t give guff about virtualization.

virsh

While setting things up, I found it frequently easier to use virsh. Here’s a little note on accessing system vm’s with virsh. There are probably some important settings I’m forgetting that I changed, although I did try my best to document while I was working on it, so hopefully not too many things…

Windows Activation

Win10 will complain about activation if you set it up for your bare hardware first, but there is a workaround (1, 2) involving using dmidecode to output your information and matching it up. For me, I added something like this, which seemed to work:

<sysinfo type='smbios'>
    <bios>
      <entry name='vendor'>American Megatrends Inc</entry>
    </bios>
    <system>
      <entry name='manufacturer'>System manufacturer</entry>
      <entry name='product'>System Product Name</entry>
      <entry name='version'>System Version</entry>
      <entry name='uuid'>[YOUR_UUID]</entry>
    </system>
    <baseBoard>
      <entry name='manufacturer'>ASUSTeK COMPUTER INC.</entry>
      <entry name='product'>ROG CROSSHAIR VIII HERO (WI-FI)</entry>
      <entry name='version'>Rev X.0x</entry>
      <entry name='serial'>[YOUR_SERIAL]</entry>
    </baseBoard>
  </sysinfo>

You should use the dmidecode output to guide you on this.

CPU Topology

I used AMD μProf to help with mapping out my 3700X (and this writeup on CPU-pinning):

./AMDuProfCLI info --cpu-topology
---------------------------------------------
 Processor  NumaNode     Die    CCX      Core
---------------------------------------------
   0         0           0      0        0   
   0         0           0      0        1   
   0         0           0      0        2   
   0         0           0      0        3   
   0         0           0      0        8   
   0         0           0      0        9   
   0         0           0      0        10  
   0         0           0      0        11  
                                -------------
   0         0           0      1        4   
   0         0           0      1        5   
   0         0           0      1        6   
   0         0           0      1        7   
   0         0           0      1        12  
   0         0           0      1        13  
   0         0           0      1        14  
   0         0           0      1        15  
---------------------------------------------

I ended up deciding to use one whole CCX (and 16GB of RAM) for the virtual machine:

<vcpu placement='static'>8</vcpu>
<iothreads>1</iothreads>   
<cputune>     
   <vcpupin vcpu='0' cpuset='4'/>     
   <vcpupin vcpu='1' cpuset='5'/>     
   <vcpupin vcpu='2' cpuset='6'/>     
   <vcpupin vcpu='3' cpuset='7'/>     
   <vcpupin vcpu='4' cpuset='12'/>     
   <vcpupin vcpu='5' cpuset='13'/>     
   <vcpupin vcpu='6' cpuset='14'/>     
   <vcpupin vcpu='7' cpuset='15'/>     
   <emulatorpin cpuset='0-1'/>     
   <iothreadpin iothread='1' cpuset='0-1'/>   
</cputune>

GPU and Display

I wasn’t too worried about this initially since I basically only wanted to use this for VR, so I just need to be able to launch SteamVR, but early on I bumped up the QXL display adapter’s memory to 32768 in the XML so that I was able to run at a higher resolution.

But, because anything that runs on that primary display chugs and it was giving me issues w/ SteamVR Desktop, I ended up disabling the QXL display in Windows entirely and if I have a screen plugged into my 1080Ti it seems to be happier (I tried Looking Glass but it wasn’t very stable and ended up adding a KVM that works well instead).

Valve Index Setup

I used a combination of dmesg and lsusb to track down the USB devices that comprise the Valve Index. For those interested:

devices = [
  ('28de','2613'), # Valve Software Hub
  ('0424','5744'), # Standard Microsystems Hub
  ('0424','2744'), # Standard Microsystems Hub
  ('28de','2102'), # Valve Software VR Radio & HMD Mic
  ('28de','2300'), # Valve Software Valve Index HMD LHR
  ('0424','2740'), # Standard Microsystems Hub Controller
  ('28de','2400'), # Valve Software Etron Technology 3D Camera
]

Before digging out my USB PCI adapters I wrote a script to create custom udev rules to do a virsh attach-device with appropriate hostdev files, but this didn’t work. I didn’t want to go down the ACS path (1, 2) for USB, which looked pretty hairy, although I did map out the C8H’s USB controllers (using lsusb -t and plugging things in sequentially, included here for interest:

Rear     Front
5 5 3 3  3
5 5 3 3  3
1 1      4
2 1

# Top Left
480M
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

# Back USB-C
10G
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

# Bottom Left
480M
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  Bus 001 Device 004: ID 8087:0029 Intel Corp. 
  Bus 001 Device 005: ID 0b05:18f3 ASUSTek Computer, Inc. 
  Bus 001 Device 003: ID 05e3:0610 Genesys Logic, Inc. 4-port hub

# Front Panel USB-C
10G
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
  5G
  Bus 004 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub

# Front Panel USB-A, Top Right
480M
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  Bus 003 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
  Bus 003 Device 005: ID 2516:0004 Cooler Master Co., Ltd. Storm QuickFire Rapid Mechanical Keyboard
  Bus 003 Device 006: ID 046d:c53f Logitech, Inc. 

Instead, I used a separate USB card. I had two, and started with a Fresco Logic card – this actually worked after adding it to VFIO, however it doesn’t reset properly so requires a hard reboot to cycle after a VM start/stop. However, I had a Renasas chipset card as well, which worked fine (here’s a guide, and another discussion) – if you’re looking for a USB card, just do a search for uPD720202 on Amazon or whatever your preferred retailer is (although I am using a 2-port uPD720201 without issues).

Tuning

I was getting some stutters originally (this might have been because I didn’t vfio the USB card originally) but I also went ahead and added <ioapic driver='kvm'/> per this note.
I didn’t do any MSI tuning or switching to invtsc since things seem to be running OK at this point.

KVM Setup

I got an AV Access HDMI 2.0 4 port KVM (4K60P YUV444 18Gbps) for $80 which seems to work pretty well. I had some issues with QXL being detected as the primary monitor (maybe due to lag on switching or something) though and in the end, I used virsh to manually remove the QXL adapter entirely from my XML config, which seems to be fine and solves that (a bunch of VR titles get unhappy if they are not connected to the primary display) – note that <graphics> needs to also be removed along with <video> otherwise a video item gets readded (see here).

TODO: libvirtd startup

For whatever reason, on bootup the systemd service has issues so before I run my VM, I need to systemctl restart libvirtd (which runs fine). This could probably be solved by swapping around the service order or something…

TODO: Dual Boot

Somewhere in between where I started and ended, dual booting got borked (Windows says it can’t find the boot media when booting) – I suspect this might have to do with when I installed some virtio drivers trying to get virtio-scsi to work or maybe an UEFI issue. I will need to get motivated enough to poke sometime, as Ryzen Master nor Thaiphoon Burner work in the vm.

Besides the last to niggles mentioned, this setup has worked pretty darn well for the past couple months. I noticed that recently the Windows Activation message popped up, I’m not quite sure what’s up with that and might need to re-register it at some point, but besides that, and being a bit of a technical workout, after the initial setup, this has been surprisingly trouble free. Fingers crossed (and a potential addendum) after the next BIOS upgrade.