You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by James Steele <js...@gmail.com> on 2021/10/01 15:01:04 UTC
AMD graphics PCI passthrough possible?
Hi all,
we have added some Ubuntu 20.04 hosts which have an AMD ATI Radeon Pro WX 5100 fitted inside.
We would like to passthrough the Radeon PCI device to KVM guests.
IOMMU has been setup correctly and the Radeon card is showing as a VFIO-PCI device. lspci -k shows:
d9:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 5100]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 5100]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
d9:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
CloudStack has native NVIDIA support, where Compute Offerings can have NVIDIA devices specified.
Is support for AMD coming? Has anyone managed to get AMD passthrough working?
Does anything need to be specified perhaps in: /etc/cloudstack/agent/agent.properties ???
Thanks, Jim
Re: AMD graphics PCI passthrough possible?
Posted by Ivan Kudryavtsev <iv...@bw-sw.com>.
Take a look at this mr:
https://github.com/apache/cloudstack/pull/3839/files
сб, 2 окт. 2021 г., 10:09 Ivan Kudryavtsev <iv...@bw-sw.com>:
> Hi, it can be done with cloudstack agent hooks implemented in groovy, but
> takes some coding and design.
>
> пт, 1 окт. 2021 г., 22:04 James Steele <js...@gmail.com>:
>
>> Hi all,
>>
>> we have added some Ubuntu 20.04 hosts which have an AMD ATI Radeon Pro WX
>> 5100 fitted inside.
>> We would like to passthrough the Radeon PCI device to KVM guests.
>>
>> IOMMU has been setup correctly and the Radeon card is showing as a
>> VFIO-PCI device. lspci -k shows:
>>
>>
>> d9:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
>> Ellesmere [Radeon Pro WX 5100]
>> Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere
>> [Radeon Pro WX 5100]
>> Kernel driver in use: vfio-pci
>> Kernel modules: amdgpu
>> d9:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere
>> HDMI Audio [Radeon RX 470/480 / 570/580/590]
>> Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI
>> Audio [Radeon RX 470/480 / 570/580/590]
>> Kernel driver in use: vfio-pci
>> Kernel modules: snd_hda_intel
>>
>>
>> CloudStack has native NVIDIA support, where Compute Offerings can have
>> NVIDIA devices specified.
>> Is support for AMD coming? Has anyone managed to get AMD passthrough
>> working?
>>
>> Does anything need to be specified perhaps in:
>> /etc/cloudstack/agent/agent.properties ???
>>
>> Thanks, Jim
>>
>
Re: AMD graphics PCI passthrough possible?
Posted by Ivan Kudryavtsev <iv...@bw-sw.com>.
Hi, it can be done with cloudstack agent hooks implemented in groovy, but
takes some coding and design.
пт, 1 окт. 2021 г., 22:04 James Steele <js...@gmail.com>:
> Hi all,
>
> we have added some Ubuntu 20.04 hosts which have an AMD ATI Radeon Pro WX
> 5100 fitted inside.
> We would like to passthrough the Radeon PCI device to KVM guests.
>
> IOMMU has been setup correctly and the Radeon card is showing as a
> VFIO-PCI device. lspci -k shows:
>
>
> d9:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
> Ellesmere [Radeon Pro WX 5100]
> Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere
> [Radeon Pro WX 5100]
> Kernel driver in use: vfio-pci
> Kernel modules: amdgpu
> d9:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere
> HDMI Audio [Radeon RX 470/480 / 570/580/590]
> Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI
> Audio [Radeon RX 470/480 / 570/580/590]
> Kernel driver in use: vfio-pci
> Kernel modules: snd_hda_intel
>
>
> CloudStack has native NVIDIA support, where Compute Offerings can have
> NVIDIA devices specified.
> Is support for AMD coming? Has anyone managed to get AMD passthrough
> working?
>
> Does anything need to be specified perhaps in:
> /etc/cloudstack/agent/agent.properties ???
>
> Thanks, Jim
>
Re: RE: AMD graphics PCI passthrough possible?
Posted by James Steele <js...@gmail.com>.
This is the XML from a running Windows Server 2019 guest that works with KVM on the Ubuntu 20.04 host. It was created directly with VMM - not CloudStack.
After windows is installed, virtio-win-0.1.208.iso is run, rebooted, then the AMD Radeon WX5100 driver package is installed.
Everything works fine, the Radeon WX5100 is available, and the machine can reboot fine:
root@macccg04:~# virsh dumpxml win2k19 > ok.xml
root@macccg04:~# cat ok.xml
<domain type='kvm' id='96'>
<name>win2k19</name>
<uuid>67e14136-8bcd-43d2-8e49-da0e2435785c</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/2k19"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>4</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='randomid'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
</features>
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>Cascadelake-Server</model>
<vendor>Intel</vendor>
<feature policy='require' name='ss'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='tsc_adjust'/>
<feature policy='require' name='umip'/>
<feature policy='require' name='pku'/>
<feature policy='require' name='md-clear'/>
<feature policy='require' name='stibp'/>
<feature policy='require' name='arch-capabilities'/>
<feature policy='require' name='xsaves'/>
<feature policy='require' name='ibpb'/>
<feature policy='require' name='ibrs'/>
<feature policy='require' name='amd-stibp'/>
<feature policy='require' name='amd-ssbd'/>
<feature policy='require' name='rdctl-no'/>
<feature policy='require' name='ibrs-all'/>
<feature policy='require' name='skip-l1dfl-vmentry'/>
<feature policy='require' name='mds-no'/>
<feature policy='require' name='pschange-mc-no'/>
<feature policy='require' name='tsx-ctrl'/>
<feature policy='disable' name='hle'/>
<feature policy='disable' name='rtm'/>
<feature policy='disable' name='mpx'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/win2k19.qcow2' index='2'/>
<backingStore/>
<target dev='sda' bus='sata'/>
<alias name='sata0-0-0'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/home/jim/Downloads/virtio-win-0.1.208.iso' index='1'/>
<backingStore/>
<target dev='sdb' bus='sata'/>
<readonly/>
<alias name='sata0-0-1'/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='sata' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<alias name='pci.3'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<alias name='pci.4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<alias name='pci.5'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<alias name='pci.6'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<alias name='pci.7'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x17'/>
<alias name='pci.8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x18'/>
<alias name='pci.9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x19'/>
<alias name='pci.10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/>
<alias name='pci.11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/>
<alias name='pci.12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:f3:17:1d'/>
<source bridge='cloudbr0'/>
<target dev='vnet0'/>
<model type='e1000e'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0' state='connected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<alias name='input0'/>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'>
<alias name='input1'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input2'/>
</input>
<graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
<image compression='off'/>
</graphics>
<sound model='ich9'>
<alias name='sound0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</hostdev>
<redirdev bus='usb' type='spicevmc'>
<alias name='redir0'/>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<alias name='redir1'/>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label>
<imagelabel>+0:+0</imagelabel>
</seclabel>
</domain>
Now here is the XML of a CloudStack created Windows Server 2019 guest (same host), which has extraconfig-1 & 2 added to it as suggested here: https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/ so that the AMD card is visible:
extraconfig-1
<features>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vendor_id state='on' value='randomid'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
</features>
and:
extraconfig-2
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x1'/>
</hostdev>
</devices>
Once the extraconfig has been added (which is based on the XML from the top working guest created in VMM), the AMD Radeon WX5100 devices show up correctly.
Again, after windows is installed, virtio-win-0.1.208.iso is run, rebooted, then the AMD Radeon WX5100 driver package is installed.
But AS SOON AS the latest AMD driver has been added, the guest will hang when rebooted. Only one guest is ever running on the host.
So the latest XML for this running but hung Windows guest looks like this:
root@macccg04:~# virsh dumpxml i-2-21-VM > bust.xml
root@macccg04:~# cat bust.xml
<domain type='kvm' id='95'>
<name>i-2-21-VM</name>
<uuid>2385b7c6-ba7f-45c9-9386-6879cba23776</uuid>
<description>Windows PV</description>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>4</vcpu>
<cputune>
<shares>4000</shares>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>2385b7c6-ba7f-45c9-9386-6879cba23776</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8096'/>
<vendor_id state='on' value='randomid'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
</features>
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>qemu64</model>
<topology sockets='1' cores='4' threads='1'/>
<feature policy='require' name='x2apic'/>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='lahf_lm'/>
<feature policy='disable' name='svm'/>
</cpu>
<clock offset='localtime'>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/mnt/1073fdbe-4b85-3727-84be-cfb6b00891f9/c52069b0-c4eb-4a62-9bcc-ff3f6a00faea' index='2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<serial>c52069b0c4eb4a629bcc</serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/mnt/5143c7c8-3f38-396e-8f51-9b818a406139/204-2-2f4adced-d319-3233-a5f1-e7b2a1b0ad9c.iso' index='1'/>
<backingStore/>
<target dev='hdc' bus='ide'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='piix3-uhci'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<alias name='virtio-serial0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='02:00:14:ea:00:0e'/>
<source bridge='brens4f0np0-633'/>
<bandwidth>
<inbound average='25600' peak='25600'/>
<outbound average='25600' peak='25600'/>
</bandwidth>
<target dev='vnet0'/>
<model type='virtio'/>
<link state='up'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/i-2-21-VM.org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
<alias name='channel0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<alias name='input0'/>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'>
<alias name='input1'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input2'/>
</input>
<graphics type='vnc' port='5900' autoport='yes' listen='10.250.0.24'>
<listen type='address' address='10.250.0.24'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</hostdev>
<watchdog model='i6300esb' action='none'>
<alias name='watchdog0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</watchdog>
<memballoon model='none'/>
</devices>
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label>
<imagelabel>+0:+0</imagelabel>
</seclabel>
</domain>
I am guessing that I need some extra bits from the working XML config (guest created in VMM) pasting as extraconfigs into the broken guests XML (guest created in CloudStack)? Any ideas?
In the working XML I see lots (12) of these specified: <controller type='pci' index='1' model='pcie-root-port'>
- but they are not in the bottom config from cloudstack.
Also in my extraconfig-2 I specify the pci devices to be mapped to <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/> and <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x1'/>
The guest created in VMM instead has: <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> and: <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
Could that be the problem somehow?
Once extraconfig-2 is removed the windows guest will again boot normally - but obviously without the AMD Radeon WX5100.
Sorry for the longest post EVER, Jim
RE: RE: AMD graphics PCI passthrough possible?
Posted by Piotr Pisz <pi...@piszki.pl>.
Hi!
Qemu, libvirt, kernel and CS compatibility is important. You can experiment here, eg qemu-ev in CentOS 7 only works well up to a certain version. In my experience (I am the author of this manual) there is no major problem with passthrough and vgpu, since I had no problem with either Linux or Windows.
Are you sure windows keeps crashing? Maybe the graphics driver for the console is not starting correctly (rdp will work normally)?
Unfortunately, I did not deal with Radeon cards, only Nvidia, maybe there are additional things to configure here.
Regards,
Piotr
-----Original Message-----
From: Alex Mattioli <Al...@shapeblue.com>
Sent: Friday, October 8, 2021 11:58 AM
To: users@cloudstack.apache.org
Subject: RE: RE: AMD graphics PCI passthrough possible?
Hi Jamex,
Am glad the guide was of help and it works well for Linux. I'm really not sure what could be going wrong with the Windows VM though, I suggest you try installing windows directly on the physical host to see if the card drivers work at all.
Cheers
Alex
-----Original Message-----
From: James Steele <js...@gmail.com>
Sent: 08 October 2021 11:03
To: users@cloudstack.apache.org
Subject: Re: RE: AMD graphics PCI passthrough possible?
Hi, and thanks for the tips everyone.
Using https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/ as a guide, I got the Radeon Pro WX 5100 PCI id's and modified Grub on my Ubuntu 20.04 host with this:
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on kvm.ignore_msrs=1 vfio-pci.ids=1002:67c7,1002:aaf0"
The Radeon card can now be assigned easily to Linux guests by adding this in the Cloudstack WebUI (Compute, Instances, then select the Instance, Settings) instance settings:
#Name:
extraconfig-1
#Value:
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0' />
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
</hostdev>
</devices>
Linux guests work perfectly, the card is detected and is available straight away.
However with a Windows guest I have had nothing but problems!
The Radeon device will turn up OK inside the Windows guest when passed through with these 2x settings:
#Name:
extraconfig-1
#Value:
<features>
<hyperv>
<vendor_id state='on' value='randomid'/>
</hyperv>
</features>
#Name:
extraconfig-2
#Value:
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0' />
<address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x1'/>
</hostdev>
</devices>
But when I install the AMD video driver (the onboard audio installs automatically) and then reboot, the system always hangs.
I have tried multiple Windows OS versions and multiple AMD driver versions.
I don't know whether the AMD driver is broke, or this card is just not supported somehow.
If anyone has any ideas I would be pleased to hear them.
Many thanks, Jim
RE: RE: AMD graphics PCI passthrough possible?
Posted by Alex Mattioli <Al...@shapeblue.com>.
Hi Jamex,
Am glad the guide was of help and it works well for Linux. I'm really not sure what could be going wrong with the Windows VM though, I suggest you try installing windows directly on the physical host to see if the card drivers work at all.
Cheers
Alex
-----Original Message-----
From: James Steele <js...@gmail.com>
Sent: 08 October 2021 11:03
To: users@cloudstack.apache.org
Subject: Re: RE: AMD graphics PCI passthrough possible?
Hi, and thanks for the tips everyone.
Using https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/ as a guide, I got the Radeon Pro WX 5100 PCI id's and modified Grub on my Ubuntu 20.04 host with this:
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on kvm.ignore_msrs=1 vfio-pci.ids=1002:67c7,1002:aaf0"
The Radeon card can now be assigned easily to Linux guests by adding this in the Cloudstack WebUI (Compute, Instances, then select the Instance, Settings) instance settings:
#Name:
extraconfig-1
#Value:
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0' />
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
</hostdev>
</devices>
Linux guests work perfectly, the card is detected and is available straight away.
However with a Windows guest I have had nothing but problems!
The Radeon device will turn up OK inside the Windows guest when passed through with these 2x settings:
#Name:
extraconfig-1
#Value:
<features>
<hyperv>
<vendor_id state='on' value='randomid'/>
</hyperv>
</features>
#Name:
extraconfig-2
#Value:
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0' />
<address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x1'/>
</hostdev>
</devices>
But when I install the AMD video driver (the onboard audio installs automatically) and then reboot, the system always hangs.
I have tried multiple Windows OS versions and multiple AMD driver versions.
I don't know whether the AMD driver is broke, or this card is just not supported somehow.
If anyone has any ideas I would be pleased to hear them.
Many thanks, Jim
Re: RE: AMD graphics PCI passthrough possible?
Posted by James Steele <js...@gmail.com>.
Hi, and thanks for the tips everyone.
Using https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/ as a guide,
I got the Radeon Pro WX 5100 PCI id's and modified Grub on my Ubuntu 20.04 host with this:
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on kvm.ignore_msrs=1 vfio-pci.ids=1002:67c7,1002:aaf0"
The Radeon card can now be assigned easily to Linux guests by adding this in the Cloudstack WebUI (Compute, Instances, then select the Instance, Settings) instance settings:
#Name:
extraconfig-1
#Value:
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0' />
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
</hostdev>
</devices>
Linux guests work perfectly, the card is detected and is available straight away.
However with a Windows guest I have had nothing but problems!
The Radeon device will turn up OK inside the Windows guest when passed through with these 2x settings:
#Name:
extraconfig-1
#Value:
<features>
<hyperv>
<vendor_id state='on' value='randomid'/>
</hyperv>
</features>
#Name:
extraconfig-2
#Value:
<devices>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x0'/>
</source>
<alias name='hostdev0' />
<address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0xd9' slot='0x00' function='0x1'/>
</source>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x1'/>
</hostdev>
</devices>
But when I install the AMD video driver (the onboard audio installs automatically) and then reboot, the system always hangs.
I have tried multiple Windows OS versions and multiple AMD driver versions.
I don't know whether the AMD driver is broke, or this card is just not supported somehow.
If anyone has any ideas I would be pleased to hear them.
Many thanks, Jim
RE: AMD graphics PCI passthrough possible?
Posted by Alex Mattioli <Al...@shapeblue.com>.
Hi James,
If the OS can detect the card then it can be passed-through. There's a very nice article on how to achieve that: https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/
Let me know if it works for you, am quite curious about it.
Regards
Alex
-----Original Message-----
From: James Steele <js...@gmail.com>
Sent: 01 October 2021 17:01
To: users@cloudstack.apache.org
Subject: AMD graphics PCI passthrough possible?
Hi all,
we have added some Ubuntu 20.04 hosts which have an AMD ATI Radeon Pro WX 5100 fitted inside.
We would like to passthrough the Radeon PCI device to KVM guests.
IOMMU has been setup correctly and the Radeon card is showing as a VFIO-PCI device. lspci -k shows:
d9:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 5100]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 5100]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
d9:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
CloudStack has native NVIDIA support, where Compute Offerings can have NVIDIA devices specified.
Is support for AMD coming? Has anyone managed to get AMD passthrough working?
Does anything need to be specified perhaps in: /etc/cloudstack/agent/agent.properties ???
Thanks, Jim