You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Piotr Pisz <pi...@piszki.pl> on 2019/07/29 11:35:36 UTC
RE: pci passthrough with extraconfig
Hello :-)
I was able to use the extraconfig option to run vm to which through pci passthrough I exported the nvidia tesla v100 card.
We are still in the process of experimenting with vGPU (on Centos7 and kvm).
Extraconfig is a super option, everything works as planned, CloudStack is great :-)
Best regards,
Piotr
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: Friday, May 24, 2019 7:02 PM
To: users <us...@cloudstack.apache.org>; Piotr Pisz <pi...@piszki.pl>
Subject: Re: pci passthrough with extraconfig
Piotr,
yes, sorry for the confusion - in 4.12 there is only "enable.additional.vm.configuration' to be enabled - that second global setting thing (from video slides) you mentioned, is in MASTER, yet to come (probably 4.13).
4.12 should, besides specific XML chunks needed for DPDK, also support any custom XML chunk - from your last reply I believe you did manage to pass needed XML chunk to libvirt - i.e. any libvirt problems after that - you would possibly want to check on the libvirt forum etc.
Let me know of your progress,
Andrija
On Fri, 24 May 2019 at 08:29, Piotr Pisz <pi...@piszki.pl> wrote:
> Hi Andrija,
>
> Yes, I'm testing it on 4.12 and I can confirm that with the example of
> "hugepages" everything works as it should.
> I do not only understand this option, I do not see it in global
> settings even as root admin:
>
> A configurable global config 'allow.additional.vm.configuration.list.kvm'
> is available for Root admin to specify a list options that are allowed
> with this feature. with KVM No option is allowed by default.
>
> What is "option" ?
>
> I try with this config:
> <devices>
> <interface type='hostdev' managed='yes'>
> <source>
> <address type='pci' domain='0x0000' bus='0x16' slot='0x07'
> function='0x2'/>
> </source>
> </interface>
> </devices>
>
> But now I have this:
>
> LibvirtException
> org.libvirt.LibvirtException: Device 0000:16:07.2 not found: could not
> access /sys/bus/pci/devices/0000:16:07.2/config: No such file or
> directory
>
> Ehh...
> Any help is welcome :)
>
> Regards,
> Piotr
>
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, May 23, 2019 9:53 PM
> To: users <us...@cloudstack.apache.org>; Piotr Pisz <pi...@piszki.pl>
> Subject: Re: pci passthrough with extraconfig
>
> You are testing this on 4.12?
> You have set global config enable.additional.vm.configuration to TRUE ?
>
> Can you try with more simple XML changes - to confirm your test ? I.e.
> must huge pages from that article (part of DPDK extra config) - later
> do "virsh dumpxml" to confirm
>
> On Thu, 23 May 2019 at 10:34, Piotr Pisz <pi...@piszki.pl> wrote:
>
> > I'm coming back with the topic
> >
> > I try to passthrough a network device (eno3) to VM on CentOS 7 based
> > on (
> > https://www.linuxtopia.org/online_books/rhel6/rhel_6_virtualization/
> > rh el_6_virtualization_chap-Virtualization-PCI_passthrough.html
> > )
> >
> > Identify device:
> > ethtool -i eno3 (bus-info: 0000:16:00.2) virsh nodedev-dumpxml
> > pci_0000_16_00_2
> >
> > Make config:
> > <hostdev mode='subsystem' type='pci' managed='yes'>
> > <source>
> > <address domain='0x0000' bus='0x16' slot='0x00' function='0x2'/>
> > </source>
> > </hostdev>
> >
> > Update VM:
> > update virtualmachine id=c264cecb-b970-4cf8-b3c2-7a54763b55ad
> >
> extraconfig=%3Chostdev%20mode%3D%27subsystem%27%20type%3D%27pci%27%20m
> anaged%3D%27yes%27%3E%0A%20%20%3Csource%3E%0A%20%20%20%20%20%2{
> > "virtualmachine": {...
> > 6%27%20slot%3D%270x00%27%20function%3D%270x2%27%2F%3E%0A%20%20%3C%2F
> > so
> > urce%3E%0A%3C%2Fhostdev%3E
> >
> > "extraconfig-1": "<hostdev mode='subsystem' type='pci' managed='yes'>\n
> > <source>\n <address domain='0x0000' bus='0x16' slot='0x00'
> > function='0x2'/>\n </source>\n</hostdev>",
> >
> > Detach device virsh nodedev-dettach pci_0000_16_00_2 on host and
> > reboot VM
> >
> > And... not see this pci on VM :-(
> >
> > The subject is new to me, what am I doing wrong? Is this path correct?
> > How do you verify that this device is connected to vm?
> >
> > Best regards,
> > Piotr
> >
> >
> >
> >
> > -----Original Message-----
> > From: Andrija Panic <an...@shapeblue.com>
> > Sent: Friday, April 5, 2019 11:42 AM
> > To: users@cloudstack.apache.org; piotr@piszki.pl
> > Subject: RE: pci passthrough
> >
> > Hi Piotr,
> >
> > Have a look at a video from last London User Group that explains
> > (beside other new things) passing extra configuration flags for KVM
> > VM, which might give you an idea how you can achieve PCI passthrough:
> > https://www.shapeblue.com/cloudstack-european-user-group-cseug-round
> > up
> > -london-march-2019/
> > - video from Bobby Stoyanov.
> >
> > Basically, and admin needs to approve what kind of additional XML
> > stuff can be sent (for security reasons), then user can consume that
> > (i.e. pass additional URL encoded XML chunk).
> >
> > Best,
> > Andrija
> >
> > andrija.panic@shapeblue.com
> > www.shapeblue.com
> > Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Piotr Pisz <pi...@piszki.pl>
> > Sent: 05 April 2019 08:37
> > To: users@cloudstack.apache.org
> > Subject: pci passthrough
> >
> > Hello,
> >
> > Is there a way to do pci passthrough to vm (eg. via VFIO/IOMMU)?
> >
> > On this article:
> > https://www.shapeblue.com/openvswitch-with-dpdk-support-on-cloudstac
> > k/
> >
> > There is talk of additional options sent to vm, can we use this to
> > pci passthrough?
> >
> > Regards,
> > Piotr
> >
> >
> >
> >
>
>
--
Andrija Panić
Re: pci passthrough with extraconfig
Posted by Andrija Panic <an...@gmail.com>.
Hey Piotr,
great to hear that - that's what it is there for :)
Since you love CloudStack :-) - would you mind sharing your example "code"
here (or PM me) - so we can update the documentation eventually - a real
world example like yours would be great!
Or you can also update the docs your self if you have time and will!
Cheers
On Mon, 29 Jul 2019 at 13:35, Piotr Pisz <pi...@piszki.pl> wrote:
> Hello :-)
>
> I was able to use the extraconfig option to run vm to which through pci
> passthrough I exported the nvidia tesla v100 card.
> We are still in the process of experimenting with vGPU (on Centos7 and
> kvm).
>
> Extraconfig is a super option, everything works as planned, CloudStack is
> great :-)
>
> Best regards,
> Piotr
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Friday, May 24, 2019 7:02 PM
> To: users <us...@cloudstack.apache.org>; Piotr Pisz <pi...@piszki.pl>
> Subject: Re: pci passthrough with extraconfig
>
> Piotr,
>
> yes, sorry for the confusion - in 4.12 there is only
> "enable.additional.vm.configuration' to be enabled - that second global
> setting thing (from video slides) you mentioned, is in MASTER, yet to come
> (probably 4.13).
>
> 4.12 should, besides specific XML chunks needed for DPDK, also support any
> custom XML chunk - from your last reply I believe you did manage to pass
> needed XML chunk to libvirt - i.e. any libvirt problems after that - you
> would possibly want to check on the libvirt forum etc.
>
> Let me know of your progress,
>
> Andrija
>
>
> On Fri, 24 May 2019 at 08:29, Piotr Pisz <pi...@piszki.pl> wrote:
>
> > Hi Andrija,
> >
> > Yes, I'm testing it on 4.12 and I can confirm that with the example of
> > "hugepages" everything works as it should.
> > I do not only understand this option, I do not see it in global
> > settings even as root admin:
> >
> > A configurable global config 'allow.additional.vm.configuration.list.kvm'
> > is available for Root admin to specify a list options that are allowed
> > with this feature. with KVM No option is allowed by default.
> >
> > What is "option" ?
> >
> > I try with this config:
> > <devices>
> > <interface type='hostdev' managed='yes'>
> > <source>
> > <address type='pci' domain='0x0000' bus='0x16' slot='0x07'
> > function='0x2'/>
> > </source>
> > </interface>
> > </devices>
> >
> > But now I have this:
> >
> > LibvirtException
> > org.libvirt.LibvirtException: Device 0000:16:07.2 not found: could not
> > access /sys/bus/pci/devices/0000:16:07.2/config: No such file or
> > directory
> >
> > Ehh...
> > Any help is welcome :)
> >
> > Regards,
> > Piotr
> >
> >
> >
> >
> > -----Original Message-----
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Thursday, May 23, 2019 9:53 PM
> > To: users <us...@cloudstack.apache.org>; Piotr Pisz <pi...@piszki.pl>
> > Subject: Re: pci passthrough with extraconfig
> >
> > You are testing this on 4.12?
> > You have set global config enable.additional.vm.configuration to TRUE ?
> >
> > Can you try with more simple XML changes - to confirm your test ? I.e.
> > must huge pages from that article (part of DPDK extra config) - later
> > do "virsh dumpxml" to confirm
> >
> > On Thu, 23 May 2019 at 10:34, Piotr Pisz <pi...@piszki.pl> wrote:
> >
> > > I'm coming back with the topic
> > >
> > > I try to passthrough a network device (eno3) to VM on CentOS 7 based
> > > on (
> > > https://www.linuxtopia.org/online_books/rhel6/rhel_6_virtualization/
> > > rh el_6_virtualization_chap-Virtualization-PCI_passthrough.html
> > > )
> > >
> > > Identify device:
> > > ethtool -i eno3 (bus-info: 0000:16:00.2) virsh nodedev-dumpxml
> > > pci_0000_16_00_2
> > >
> > > Make config:
> > > <hostdev mode='subsystem' type='pci' managed='yes'>
> > > <source>
> > > <address domain='0x0000' bus='0x16' slot='0x00' function='0x2'/>
> > > </source>
> > > </hostdev>
> > >
> > > Update VM:
> > > update virtualmachine id=c264cecb-b970-4cf8-b3c2-7a54763b55ad
> > >
> > extraconfig=%3Chostdev%20mode%3D%27subsystem%27%20type%3D%27pci%27%20m
> > anaged%3D%27yes%27%3E%0A%20%20%3Csource%3E%0A%20%20%20%20%20%2{
> > > "virtualmachine": {...
> > > 6%27%20slot%3D%270x00%27%20function%3D%270x2%27%2F%3E%0A%20%20%3C%2F
> > > so
> > > urce%3E%0A%3C%2Fhostdev%3E
> > >
> > > "extraconfig-1": "<hostdev mode='subsystem' type='pci' managed='yes'>\n
> > > <source>\n <address domain='0x0000' bus='0x16' slot='0x00'
> > > function='0x2'/>\n </source>\n</hostdev>",
> > >
> > > Detach device virsh nodedev-dettach pci_0000_16_00_2 on host and
> > > reboot VM
> > >
> > > And... not see this pci on VM :-(
> > >
> > > The subject is new to me, what am I doing wrong? Is this path correct?
> > > How do you verify that this device is connected to vm?
> > >
> > > Best regards,
> > > Piotr
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Andrija Panic <an...@shapeblue.com>
> > > Sent: Friday, April 5, 2019 11:42 AM
> > > To: users@cloudstack.apache.org; piotr@piszki.pl
> > > Subject: RE: pci passthrough
> > >
> > > Hi Piotr,
> > >
> > > Have a look at a video from last London User Group that explains
> > > (beside other new things) passing extra configuration flags for KVM
> > > VM, which might give you an idea how you can achieve PCI passthrough:
> > > https://www.shapeblue.com/cloudstack-european-user-group-cseug-round
> > > up
> > > -london-march-2019/
> > > - video from Bobby Stoyanov.
> > >
> > > Basically, and admin needs to approve what kind of additional XML
> > > stuff can be sent (for security reasons), then user can consume that
> > > (i.e. pass additional URL encoded XML chunk).
> > >
> > > Best,
> > > Andrija
> > >
> > > andrija.panic@shapeblue.com
> > > www.shapeblue.com
> > > Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Piotr Pisz <pi...@piszki.pl>
> > > Sent: 05 April 2019 08:37
> > > To: users@cloudstack.apache.org
> > > Subject: pci passthrough
> > >
> > > Hello,
> > >
> > > Is there a way to do pci passthrough to vm (eg. via VFIO/IOMMU)?
> > >
> > > On this article:
> > > https://www.shapeblue.com/openvswitch-with-dpdk-support-on-cloudstac
> > > k/
> > >
> > > There is talk of additional options sent to vm, can we use this to
> > > pci passthrough?
> > >
> > > Regards,
> > > Piotr
> > >
> > >
> > >
> > >
> >
> >
>
> --
>
> Andrija Panić
>
>
--
Andrija Panić