You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by JeanPaul van der Mijle <jp...@hyperai.ai> on 2023/02/12 13:44:39 UTC

PCI-E Passthrough

Hi all,

Does anyone have a good guide on adding a PCI-E card (A Mellanox Connect-X 5 VF interface) to cloudstack?
I had this all running before in stock Qemu, and all settings are still present for the card and VF's, I just need to know how the management sees that I have additional cards that I have available to assign to VMs.

I tried adding:
pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1

However, after restarting the agent, I noticed that it added escape characters:
pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|100GE,c1\:01.1

Not sure if this is fine, but so far it was the only thing I could find back on google, but this is 10 years ago. I suspect this is no longer sufficient.
After restarting the agent, I do not see any changes unfortunately.

Thanks!

JeanPaul


Re: PCI-E Passthrough

Posted by Wei ZHOU <us...@gmail.com>.
Hi JeanPaul,

Just for your information, CloudStack starts the VMs from a XML file which
is different from "virsh dumpxml".

You can get the VM definition XML in agent.log if you have enabled DEBUG in
log4j. If not, you can do it by the following command

cp /etc/cloudstack/agent/log4j-cloud.xml
/etc/cloudstack/agent/log4j-cloud.xml.orig && \
sed -i "s/INFO/DEBUG/g" /etc/cloudstack/agent/log4j-cloud.xml && \
systemctl restart cloudstack-agent

-Wei

On Thu, 16 Feb 2023 at 11:42, JeanPaul van der Mijle <
jp.vandermijle@hyperai.ai> wrote:

> Hi all,
>
> Another update, think we can close this anyway now, if there are other
> things going south it is most likely somewhere else.
>
> In the tests of KVM/Qemu before Cloudstack I could give any address it
> needs to be exposed to. In Cloudstack you would need to start on domain 0,
> bus 0, slot 0 and function 0 and count up the function for each device.
> Doing it differently will not expose it to the vm for me.
>
> I should take a look further to better understand why this difference as
> I'd love to understand and also document this very valuable information for
> myself.
>
> Thanks again!
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp...@hyperai.ai>
> Sent: Wednesday, February 15, 2023 4:29 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi All,
>
> Adding some additional information, as for custom configuration it now
> works.
> I did have some trouble first with getting the VM boot, as it apparently
> needs specific settings for the pci bus/slot etc. I took over the config as
> is from an VM where it worked with before, before I installed cloudstack.
> Now I have found that out, I am stuck with the following:
>
> When starting the VM, I see in syslog on the host/hypervisor that it is
> enabling the device for passthrough:
> Feb 15 16:20:48 gc-hs1 kernel: [18145.274514] vfio-pci 0000:81:00.2:
> enabling device (0000 -> 0002)
>
> I also can see it in the log of the VM, it uses the parameters:
> -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:00:34:00:00:f6,bus=pci.0,addr=0x3
> \
>
> When doing virsh edit on the domain, I get:
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <driver name='vfio'/>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
>       </source>
>       <alias name='hostdev0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x0c'
> function='0x1'/>
>     </hostdev>
>
>
> Only thing I never added was driver name, I am a bit suspicious on that
> but probably has it's reasons. It worked on stand alone virsh+qemu setup
> but there I never had to give up the driver.
>
> Is there some additional configuration that is required?
>
> For now I test with 1 device, but eventually there should be 4 Mellanox
> VF's being exposed/passed to the VM.
>
> Thank you yet again,
>
> JeanPaul
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Wednesday, February 15, 2023 10:42 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> You are welcome. Good to know it works.
>
> -Wei
>
>
> On Wednesday, 15 February 2023, JeanPaul van der Mijle <
> jp.vandermijle@hyperai.ai> wrote:
>
> > Hi All,
> >
> > Nevermind, figured out, it is very important on the alphabetic order when
> > signing the request.
> >
> > Thanks anyways!
> >
> > JeanPaul
> > ________________________________
> > From: JeanPaul van der Mijle <jp...@hyperai.ai>
> > Sent: Wednesday, February 15, 2023 10:05 AM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: PCI-E Passthrough
> >
> > Hi Wei, All,
> >
> > I am currently stuck with the API call, I manage to sign requests as
> > listUsers and plain updateVirtualMachine with id works, but as soon as I
> > add the extraconfig parameter, it keeps complaining it is not correct.
> This
> > is so far with GET request only, I have not yet tried POST, as I haven't
> > taken a look if I should do every value with post (command, id,
> > extraconfig, response and signature?) or just the extraconfig and how
> > signing would go then.
> >
> > I have tried to sign the request with: "command=updatevirtualmachine&
> > id=xxxx&extraconfig=urlencodedconfig"
> > both fully lowercase, as well lowercase only the command.
> >
> > I also tried without adding extraconfig to the signature generation but
> > they all end up with:
> >
> >     ["errorcode"]=>
> >     int(401)
> >     ["errortext"]=>
> >     string(58) "unable to verify user credentials and/or request
> signature"
> >
> > Currently using PHP to do the signatures and it looks like as following:
> >
> > <?php
> > $baseurl = "http://localhost:8080/client/api?";
> >
> > $response = "response=json";
> >
> > $vmid = "id=6f695b14-71b4-4deb-bc4c-851580324f91";
> >
> > $apikey = "apikey=<apikey>";
> > $secretkey = "<secret>";
> > $configxml = <<<EOD
> > <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
> >       </source>
> >       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> > function='0x0' multifunction='on'/>
> >     </hostdev>
> >     <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
> >       </source>
> >       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> > function='0x0' multifunction='on'/>
> >     </hostdev>
> >     <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
> >       </source>
> >       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> > function='0x1'/>
> >     </hostdev>
> >     <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
> >       </source>
> >       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> > function='0x1'/>
> >     </hostdev>
> >     <memballoon model='virtio'>
> >       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > function='0x0'/>
> >     </memballoon>
> >     <rng model='virtio'>
> >       <backend model='random'>/dev/urandom</backend>
> >       <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> > function='0x0'/>
> >     </rng>
> > EOD;
> >
> > $extraconfig = "extraconfig=" . urlencode($configxml);
> >
> > $command = "command=updateVirtualMachine&".$vmid;
> >
> > $hash = hash_hmac("sha1",strtolower($apikey . "&" . $command) ."&" .
> > $extraconfig .  "&" . $response,$secretkey, true);
> > $base64encoded = base64_encode($hash);
> > $signature = "signature=" . urlencode($base64encoded);
> >
> > $link = $baseurl  . $apikey . "&" . $command . "&" . $extraconfig . "&"
> > .$response . "&" . $signature;
> >
> > $cURLConnection = curl_init();
> >
> > curl_setopt($cURLConnection, CURLOPT_URL, $link);
> > curl_setopt($cURLConnection, CURLOPT_RETURNTRANSFER, true);
> >
> > $json_output = curl_exec($cURLConnection);
> > curl_close($cURLConnection);
> >
> > $jsonArrayResponse = json_decode($json_output,true);
> >
> > var_dump($jsonArrayResponse);
> >
> >
> > The URL end up looking like this:
> >
> > http://localhost:8080/client/api?apikey=<Apikey>&command=
> > updateVirtualMachine&id=<vmid>&extraconfig=%3Chostdev+mode%
> > 3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%
> > 3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%
> > 270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%
> > 270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%
> > 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> > 270x07%27+slot%3D%270x00%27+function%3D%270x0%27+
> > multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++
> > ++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> > 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> > 3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%
> > 270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%
> > 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> > 27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x0%
> > 27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%
> > 0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> > 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> > 3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%
> > 270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%
> > 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> > 27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x1%
> > 27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%
> > 27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++
> > ++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%
> > 27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x6%
> > 27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%
> > 27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%
> > 270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%
> > 3E%0A++++%3Cmemballoon+model%3D%27virtio%27%3E%0A++++++%
> > 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> > 270x04%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A+
> > +++%3C%2Fmemballoon%3E%0A++++%3Crng+model%3D%27virtio%27%3E%
> > 0A++++++%3Cbackend+model%3D%27random%27%3E%2Fdev%
> > 2Furandom%3C%2Fbackend%3E%0A++++++%3Caddress+type%3D%27pci%
> > 27+domain%3D%270x0000%27+bus%3D%270x05%27+slot%3D%270x00%
> > 27+function%3D%270x0%27%2F%3E%0A++++%3C%2Frng%3E&response=
> > json&signature=<signature>
> >
> > Thanks,
> >
> > JeanPaul
> > ________________________________
> > From: JeanPaul van der Mijle <jp...@hyperai.ai>
> > Sent: Sunday, February 12, 2023 6:41 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: PCI-E Passthrough
> >
> > Hi Wei,
> >
> > Thanks, going to take a deeper look into this.
> >
> > I assume I will have to add the list as following, to allow the XML below
> > to be accepted by the API when I submit this encoded?
> >
> > I set this to be allowed for KVM: domain,devices,hostdev,source,
> > address,memballoon,rng,backend
> >
> > It should reflect this XML part:
> >
> > <domain>
> >   <device>
> >     <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
> >       </source>
> >       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> > function='0x0' multifunction='on'/>
> >     </hostdev>
> >     <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
> >       </source>
> >       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> > function='0x0' multifunction='on'/>
> >     </hostdev>
> >     <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
> >       </source>
> >       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> > function='0x1'/>
> >     </hostdev>
> >     <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
> >       </source>
> >       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> > function='0x1'/>
> >     </hostdev>
> >     <memballoon model='virtio'>
> >       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> > function='0x0'/>
> >     </memballoon>
> >     <rng model='virtio'>
> >       <backend model='random'>/dev/urandom</backend>
> >       <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> > function='0x0'/>
> >     </rng>
> >   </device>
> > </domain>
> >
> > I only need to figure out if domain and device are required or not but I
> > will see when I test it out.
> >
> > Thanks so far!
> >
> > JeanPaul
> > ________________________________
> > From: Wei ZHOU <us...@gmail.com>
> > Sent: Sunday, February 12, 2023 5:23 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: PCI-E Passthrough
> >
> > I think you need to add additional configuration to vm.
> > Refer to
> > https://www.shapeblue.com/cloudstack-feature-first-look-
> > enable-sending-of-arbitrary-configuration-data-to-vms/
> >
> > -Wei
> >
> >
> >
> > On Sunday, 12 February 2023, JeanPaul van der Mijle <
> > jp.vandermijle@hyperai.ai> wrote:
> >
> > > Hi all,
> > >
> > > Does anyone have a good guide on adding a PCI-E card (A Mellanox
> > Connect-X
> > > 5 VF interface) to cloudstack?
> > > I had this all running before in stock Qemu, and all settings are still
> > > present for the card and VF's, I just need to know how the management
> > sees
> > > that I have additional cards that I have available to assign to VMs.
> > >
> > > I tried adding:
> > > pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> > > 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> > > 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> > > 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
> > >
> > > However, after restarting the agent, I noticed that it added escape
> > > characters:
> > > pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> > > 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> > > 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> > > 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|
> > 100GE,c1\:01.1
> > >
> > > Not sure if this is fine, but so far it was the only thing I could find
> > > back on google, but this is 10 years ago. I suspect this is no longer
> > > sufficient.
> > > After restarting the agent, I do not see any changes unfortunately.
> > >
> > > Thanks!
> > >
> > > JeanPaul
> > >
> > >
> >
>

Re: PCI-E Passthrough

Posted by JeanPaul van der Mijle <jp...@hyperai.ai>.
Hi all,

Another update, think we can close this anyway now, if there are other things going south it is most likely somewhere else.

In the tests of KVM/Qemu before Cloudstack I could give any address it needs to be exposed to. In Cloudstack you would need to start on domain 0, bus 0, slot 0 and function 0 and count up the function for each device. Doing it differently will not expose it to the vm for me.

I should take a look further to better understand why this difference as I'd love to understand and also document this very valuable information for myself.

Thanks again!

JeanPaul
________________________________
From: JeanPaul van der Mijle <jp...@hyperai.ai>
Sent: Wednesday, February 15, 2023 4:29 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

Hi All,

Adding some additional information, as for custom configuration it now works.
I did have some trouble first with getting the VM boot, as it apparently needs specific settings for the pci bus/slot etc. I took over the config as is from an VM where it worked with before, before I installed cloudstack. Now I have found that out, I am stuck with the following:

When starting the VM, I see in syslog on the host/hypervisor that it is enabling the device for passthrough:
Feb 15 16:20:48 gc-hs1 kernel: [18145.274514] vfio-pci 0000:81:00.2: enabling device (0000 -> 0002)

I also can see it in the log of the VM, it uses the parameters:
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:00:34:00:00:f6,bus=pci.0,addr=0x3 \

When doing virsh edit on the domain, I get:
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x1'/>
    </hostdev>


Only thing I never added was driver name, I am a bit suspicious on that but probably has it's reasons. It worked on stand alone virsh+qemu setup but there I never had to give up the driver.

Is there some additional configuration that is required?

For now I test with 1 device, but eventually there should be 4 Mellanox VF's being exposed/passed to the VM.

Thank you yet again,

JeanPaul
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Wednesday, February 15, 2023 10:42 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

You are welcome. Good to know it works.

-Wei


On Wednesday, 15 February 2023, JeanPaul van der Mijle <
jp.vandermijle@hyperai.ai> wrote:

> Hi All,
>
> Nevermind, figured out, it is very important on the alphabetic order when
> signing the request.
>
> Thanks anyways!
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp...@hyperai.ai>
> Sent: Wednesday, February 15, 2023 10:05 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi Wei, All,
>
> I am currently stuck with the API call, I manage to sign requests as
> listUsers and plain updateVirtualMachine with id works, but as soon as I
> add the extraconfig parameter, it keeps complaining it is not correct. This
> is so far with GET request only, I have not yet tried POST, as I haven't
> taken a look if I should do every value with post (command, id,
> extraconfig, response and signature?) or just the extraconfig and how
> signing would go then.
>
> I have tried to sign the request with: "command=updatevirtualmachine&
> id=xxxx&extraconfig=urlencodedconfig"
> both fully lowercase, as well lowercase only the command.
>
> I also tried without adding extraconfig to the signature generation but
> they all end up with:
>
>     ["errorcode"]=>
>     int(401)
>     ["errortext"]=>
>     string(58) "unable to verify user credentials and/or request signature"
>
> Currently using PHP to do the signatures and it looks like as following:
>
> <?php
> $baseurl = "http://localhost:8080/client/api?";
>
> $response = "response=json";
>
> $vmid = "id=6f695b14-71b4-4deb-bc4c-851580324f91";
>
> $apikey = "apikey=<apikey>";
> $secretkey = "<secret>";
> $configxml = <<<EOD
> <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>     </memballoon>
>     <rng model='virtio'>
>       <backend model='random'>/dev/urandom</backend>
>       <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
>     </rng>
> EOD;
>
> $extraconfig = "extraconfig=" . urlencode($configxml);
>
> $command = "command=updateVirtualMachine&".$vmid;
>
> $hash = hash_hmac("sha1",strtolower($apikey . "&" . $command) ."&" .
> $extraconfig .  "&" . $response,$secretkey, true);
> $base64encoded = base64_encode($hash);
> $signature = "signature=" . urlencode($base64encoded);
>
> $link = $baseurl  . $apikey . "&" . $command . "&" . $extraconfig . "&"
> .$response . "&" . $signature;
>
> $cURLConnection = curl_init();
>
> curl_setopt($cURLConnection, CURLOPT_URL, $link);
> curl_setopt($cURLConnection, CURLOPT_RETURNTRANSFER, true);
>
> $json_output = curl_exec($cURLConnection);
> curl_close($cURLConnection);
>
> $jsonArrayResponse = json_decode($json_output,true);
>
> var_dump($jsonArrayResponse);
>
>
> The URL end up looking like this:
>
> http://localhost:8080/client/api?apikey=<Apikey>&command=
> updateVirtualMachine&id=<vmid>&extraconfig=%3Chostdev+mode%
> 3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%
> 3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%
> 270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%
> 270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%
> 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> 270x07%27+slot%3D%270x00%27+function%3D%270x0%27+
> multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++
> ++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> 3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%
> 270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%
> 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> 27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x0%
> 27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%
> 0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> 3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%
> 270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%
> 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> 27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x1%
> 27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%
> 27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++
> ++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%
> 27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x6%
> 27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%
> 27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%
> 270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%
> 3E%0A++++%3Cmemballoon+model%3D%27virtio%27%3E%0A++++++%
> 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> 270x04%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A+
> +++%3C%2Fmemballoon%3E%0A++++%3Crng+model%3D%27virtio%27%3E%
> 0A++++++%3Cbackend+model%3D%27random%27%3E%2Fdev%
> 2Furandom%3C%2Fbackend%3E%0A++++++%3Caddress+type%3D%27pci%
> 27+domain%3D%270x0000%27+bus%3D%270x05%27+slot%3D%270x00%
> 27+function%3D%270x0%27%2F%3E%0A++++%3C%2Frng%3E&response=
> json&signature=<signature>
>
> Thanks,
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp...@hyperai.ai>
> Sent: Sunday, February 12, 2023 6:41 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi Wei,
>
> Thanks, going to take a deeper look into this.
>
> I assume I will have to add the list as following, to allow the XML below
> to be accepted by the API when I submit this encoded?
>
> I set this to be allowed for KVM: domain,devices,hostdev,source,
> address,memballoon,rng,backend
>
> It should reflect this XML part:
>
> <domain>
>   <device>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>     </memballoon>
>     <rng model='virtio'>
>       <backend model='random'>/dev/urandom</backend>
>       <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
>     </rng>
>   </device>
> </domain>
>
> I only need to figure out if domain and device are required or not but I
> will see when I test it out.
>
> Thanks so far!
>
> JeanPaul
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Sunday, February 12, 2023 5:23 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> I think you need to add additional configuration to vm.
> Refer to
> https://www.shapeblue.com/cloudstack-feature-first-look-
> enable-sending-of-arbitrary-configuration-data-to-vms/
>
> -Wei
>
>
>
> On Sunday, 12 February 2023, JeanPaul van der Mijle <
> jp.vandermijle@hyperai.ai> wrote:
>
> > Hi all,
> >
> > Does anyone have a good guide on adding a PCI-E card (A Mellanox
> Connect-X
> > 5 VF interface) to cloudstack?
> > I had this all running before in stock Qemu, and all settings are still
> > present for the card and VF's, I just need to know how the management
> sees
> > that I have additional cards that I have available to assign to VMs.
> >
> > I tried adding:
> > pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> > 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> > 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> > 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
> >
> > However, after restarting the agent, I noticed that it added escape
> > characters:
> > pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> > 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> > 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> > 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|
> 100GE,c1\:01.1
> >
> > Not sure if this is fine, but so far it was the only thing I could find
> > back on google, but this is 10 years ago. I suspect this is no longer
> > sufficient.
> > After restarting the agent, I do not see any changes unfortunately.
> >
> > Thanks!
> >
> > JeanPaul
> >
> >
>

Re: PCI-E Passthrough

Posted by JeanPaul van der Mijle <jp...@hyperai.ai>.
Hi All,

Adding some additional information, as for custom configuration it now works.
I did have some trouble first with getting the VM boot, as it apparently needs specific settings for the pci bus/slot etc. I took over the config as is from an VM where it worked with before, before I installed cloudstack. Now I have found that out, I am stuck with the following:

When starting the VM, I see in syslog on the host/hypervisor that it is enabling the device for passthrough:
Feb 15 16:20:48 gc-hs1 kernel: [18145.274514] vfio-pci 0000:81:00.2: enabling device (0000 -> 0002)

I also can see it in the log of the VM, it uses the parameters:
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:00:34:00:00:f6,bus=pci.0,addr=0x3 \

When doing virsh edit on the domain, I get:
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x1'/>
    </hostdev>


Only thing I never added was driver name, I am a bit suspicious on that but probably has it's reasons. It worked on stand alone virsh+qemu setup but there I never had to give up the driver.

Is there some additional configuration that is required?

For now I test with 1 device, but eventually there should be 4 Mellanox VF's being exposed/passed to the VM.

Thank you yet again,

JeanPaul
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Wednesday, February 15, 2023 10:42 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

You are welcome. Good to know it works.

-Wei


On Wednesday, 15 February 2023, JeanPaul van der Mijle <
jp.vandermijle@hyperai.ai> wrote:

> Hi All,
>
> Nevermind, figured out, it is very important on the alphabetic order when
> signing the request.
>
> Thanks anyways!
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp...@hyperai.ai>
> Sent: Wednesday, February 15, 2023 10:05 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi Wei, All,
>
> I am currently stuck with the API call, I manage to sign requests as
> listUsers and plain updateVirtualMachine with id works, but as soon as I
> add the extraconfig parameter, it keeps complaining it is not correct. This
> is so far with GET request only, I have not yet tried POST, as I haven't
> taken a look if I should do every value with post (command, id,
> extraconfig, response and signature?) or just the extraconfig and how
> signing would go then.
>
> I have tried to sign the request with: "command=updatevirtualmachine&
> id=xxxx&extraconfig=urlencodedconfig"
> both fully lowercase, as well lowercase only the command.
>
> I also tried without adding extraconfig to the signature generation but
> they all end up with:
>
>     ["errorcode"]=>
>     int(401)
>     ["errortext"]=>
>     string(58) "unable to verify user credentials and/or request signature"
>
> Currently using PHP to do the signatures and it looks like as following:
>
> <?php
> $baseurl = "http://localhost:8080/client/api?";
>
> $response = "response=json";
>
> $vmid = "id=6f695b14-71b4-4deb-bc4c-851580324f91";
>
> $apikey = "apikey=<apikey>";
> $secretkey = "<secret>";
> $configxml = <<<EOD
> <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>     </memballoon>
>     <rng model='virtio'>
>       <backend model='random'>/dev/urandom</backend>
>       <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
>     </rng>
> EOD;
>
> $extraconfig = "extraconfig=" . urlencode($configxml);
>
> $command = "command=updateVirtualMachine&".$vmid;
>
> $hash = hash_hmac("sha1",strtolower($apikey . "&" . $command) ."&" .
> $extraconfig .  "&" . $response,$secretkey, true);
> $base64encoded = base64_encode($hash);
> $signature = "signature=" . urlencode($base64encoded);
>
> $link = $baseurl  . $apikey . "&" . $command . "&" . $extraconfig . "&"
> .$response . "&" . $signature;
>
> $cURLConnection = curl_init();
>
> curl_setopt($cURLConnection, CURLOPT_URL, $link);
> curl_setopt($cURLConnection, CURLOPT_RETURNTRANSFER, true);
>
> $json_output = curl_exec($cURLConnection);
> curl_close($cURLConnection);
>
> $jsonArrayResponse = json_decode($json_output,true);
>
> var_dump($jsonArrayResponse);
>
>
> The URL end up looking like this:
>
> http://localhost:8080/client/api?apikey=<Apikey>&command=
> updateVirtualMachine&id=<vmid>&extraconfig=%3Chostdev+mode%
> 3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%
> 3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%
> 270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%
> 270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%
> 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> 270x07%27+slot%3D%270x00%27+function%3D%270x0%27+
> multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++
> ++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> 3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%
> 270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%
> 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> 27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x0%
> 27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%
> 0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> 3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%
> 270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%
> 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> 27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x1%
> 27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%
> 27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++
> ++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%
> 27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x6%
> 27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%
> 27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%
> 270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%
> 3E%0A++++%3Cmemballoon+model%3D%27virtio%27%3E%0A++++++%
> 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> 270x04%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A+
> +++%3C%2Fmemballoon%3E%0A++++%3Crng+model%3D%27virtio%27%3E%
> 0A++++++%3Cbackend+model%3D%27random%27%3E%2Fdev%
> 2Furandom%3C%2Fbackend%3E%0A++++++%3Caddress+type%3D%27pci%
> 27+domain%3D%270x0000%27+bus%3D%270x05%27+slot%3D%270x00%
> 27+function%3D%270x0%27%2F%3E%0A++++%3C%2Frng%3E&response=
> json&signature=<signature>
>
> Thanks,
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp...@hyperai.ai>
> Sent: Sunday, February 12, 2023 6:41 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi Wei,
>
> Thanks, going to take a deeper look into this.
>
> I assume I will have to add the list as following, to allow the XML below
> to be accepted by the API when I submit this encoded?
>
> I set this to be allowed for KVM: domain,devices,hostdev,source,
> address,memballoon,rng,backend
>
> It should reflect this XML part:
>
> <domain>
>   <device>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>     </memballoon>
>     <rng model='virtio'>
>       <backend model='random'>/dev/urandom</backend>
>       <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
>     </rng>
>   </device>
> </domain>
>
> I only need to figure out if domain and device are required or not but I
> will see when I test it out.
>
> Thanks so far!
>
> JeanPaul
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Sunday, February 12, 2023 5:23 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> I think you need to add additional configuration to vm.
> Refer to
> https://www.shapeblue.com/cloudstack-feature-first-look-
> enable-sending-of-arbitrary-configuration-data-to-vms/
>
> -Wei
>
>
>
> On Sunday, 12 February 2023, JeanPaul van der Mijle <
> jp.vandermijle@hyperai.ai> wrote:
>
> > Hi all,
> >
> > Does anyone have a good guide on adding a PCI-E card (A Mellanox
> Connect-X
> > 5 VF interface) to cloudstack?
> > I had this all running before in stock Qemu, and all settings are still
> > present for the card and VF's, I just need to know how the management
> sees
> > that I have additional cards that I have available to assign to VMs.
> >
> > I tried adding:
> > pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> > 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> > 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> > 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
> >
> > However, after restarting the agent, I noticed that it added escape
> > characters:
> > pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> > 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> > 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> > 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|
> 100GE,c1\:01.1
> >
> > Not sure if this is fine, but so far it was the only thing I could find
> > back on google, but this is 10 years ago. I suspect this is no longer
> > sufficient.
> > After restarting the agent, I do not see any changes unfortunately.
> >
> > Thanks!
> >
> > JeanPaul
> >
> >
>

Re: PCI-E Passthrough

Posted by Wei ZHOU <us...@gmail.com>.
You are welcome. Good to know it works.

-Wei


On Wednesday, 15 February 2023, JeanPaul van der Mijle <
jp.vandermijle@hyperai.ai> wrote:

> Hi All,
>
> Nevermind, figured out, it is very important on the alphabetic order when
> signing the request.
>
> Thanks anyways!
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp...@hyperai.ai>
> Sent: Wednesday, February 15, 2023 10:05 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi Wei, All,
>
> I am currently stuck with the API call, I manage to sign requests as
> listUsers and plain updateVirtualMachine with id works, but as soon as I
> add the extraconfig parameter, it keeps complaining it is not correct. This
> is so far with GET request only, I have not yet tried POST, as I haven't
> taken a look if I should do every value with post (command, id,
> extraconfig, response and signature?) or just the extraconfig and how
> signing would go then.
>
> I have tried to sign the request with: "command=updatevirtualmachine&
> id=xxxx&extraconfig=urlencodedconfig"
> both fully lowercase, as well lowercase only the command.
>
> I also tried without adding extraconfig to the signature generation but
> they all end up with:
>
>     ["errorcode"]=>
>     int(401)
>     ["errortext"]=>
>     string(58) "unable to verify user credentials and/or request signature"
>
> Currently using PHP to do the signatures and it looks like as following:
>
> <?php
> $baseurl = "http://localhost:8080/client/api?";
>
> $response = "response=json";
>
> $vmid = "id=6f695b14-71b4-4deb-bc4c-851580324f91";
>
> $apikey = "apikey=<apikey>";
> $secretkey = "<secret>";
> $configxml = <<<EOD
> <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>     </memballoon>
>     <rng model='virtio'>
>       <backend model='random'>/dev/urandom</backend>
>       <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
>     </rng>
> EOD;
>
> $extraconfig = "extraconfig=" . urlencode($configxml);
>
> $command = "command=updateVirtualMachine&".$vmid;
>
> $hash = hash_hmac("sha1",strtolower($apikey . "&" . $command) ."&" .
> $extraconfig .  "&" . $response,$secretkey, true);
> $base64encoded = base64_encode($hash);
> $signature = "signature=" . urlencode($base64encoded);
>
> $link = $baseurl  . $apikey . "&" . $command . "&" . $extraconfig . "&"
> .$response . "&" . $signature;
>
> $cURLConnection = curl_init();
>
> curl_setopt($cURLConnection, CURLOPT_URL, $link);
> curl_setopt($cURLConnection, CURLOPT_RETURNTRANSFER, true);
>
> $json_output = curl_exec($cURLConnection);
> curl_close($cURLConnection);
>
> $jsonArrayResponse = json_decode($json_output,true);
>
> var_dump($jsonArrayResponse);
>
>
> The URL end up looking like this:
>
> http://localhost:8080/client/api?apikey=<Apikey>&command=
> updateVirtualMachine&id=<vmid>&extraconfig=%3Chostdev+mode%
> 3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%
> 3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%
> 270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%
> 270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%
> 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> 270x07%27+slot%3D%270x00%27+function%3D%270x0%27+
> multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++
> ++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> 3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%
> 270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%
> 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> 27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x0%
> 27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%
> 0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> 3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%
> 270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%
> 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> 27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x1%
> 27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%
> 27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++
> ++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%
> 27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x6%
> 27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%
> 27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%
> 270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%
> 3E%0A++++%3Cmemballoon+model%3D%27virtio%27%3E%0A++++++%
> 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> 270x04%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A+
> +++%3C%2Fmemballoon%3E%0A++++%3Crng+model%3D%27virtio%27%3E%
> 0A++++++%3Cbackend+model%3D%27random%27%3E%2Fdev%
> 2Furandom%3C%2Fbackend%3E%0A++++++%3Caddress+type%3D%27pci%
> 27+domain%3D%270x0000%27+bus%3D%270x05%27+slot%3D%270x00%
> 27+function%3D%270x0%27%2F%3E%0A++++%3C%2Frng%3E&response=
> json&signature=<signature>
>
> Thanks,
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp...@hyperai.ai>
> Sent: Sunday, February 12, 2023 6:41 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi Wei,
>
> Thanks, going to take a deeper look into this.
>
> I assume I will have to add the list as following, to allow the XML below
> to be accepted by the API when I submit this encoded?
>
> I set this to be allowed for KVM: domain,devices,hostdev,source,
> address,memballoon,rng,backend
>
> It should reflect this XML part:
>
> <domain>
>   <device>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x0' multifunction='on'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
>       </source>
>       <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
>     </hostdev>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
>     </memballoon>
>     <rng model='virtio'>
>       <backend model='random'>/dev/urandom</backend>
>       <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
>     </rng>
>   </device>
> </domain>
>
> I only need to figure out if domain and device are required or not but I
> will see when I test it out.
>
> Thanks so far!
>
> JeanPaul
> ________________________________
> From: Wei ZHOU <us...@gmail.com>
> Sent: Sunday, February 12, 2023 5:23 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> I think you need to add additional configuration to vm.
> Refer to
> https://www.shapeblue.com/cloudstack-feature-first-look-
> enable-sending-of-arbitrary-configuration-data-to-vms/
>
> -Wei
>
>
>
> On Sunday, 12 February 2023, JeanPaul van der Mijle <
> jp.vandermijle@hyperai.ai> wrote:
>
> > Hi all,
> >
> > Does anyone have a good guide on adding a PCI-E card (A Mellanox
> Connect-X
> > 5 VF interface) to cloudstack?
> > I had this all running before in stock Qemu, and all settings are still
> > present for the card and VF's, I just need to know how the management
> sees
> > that I have additional cards that I have available to assign to VMs.
> >
> > I tried adding:
> > pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> > 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> > 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> > 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
> >
> > However, after restarting the agent, I noticed that it added escape
> > characters:
> > pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> > 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> > 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> > 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|
> 100GE,c1\:01.1
> >
> > Not sure if this is fine, but so far it was the only thing I could find
> > back on google, but this is 10 years ago. I suspect this is no longer
> > sufficient.
> > After restarting the agent, I do not see any changes unfortunately.
> >
> > Thanks!
> >
> > JeanPaul
> >
> >
>

Re: PCI-E Passthrough

Posted by JeanPaul van der Mijle <jp...@hyperai.ai>.
Hi All,

Nevermind, figured out, it is very important on the alphabetic order when signing the request.

Thanks anyways!

JeanPaul
________________________________
From: JeanPaul van der Mijle <jp...@hyperai.ai>
Sent: Wednesday, February 15, 2023 10:05 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

Hi Wei, All,

I am currently stuck with the API call, I manage to sign requests as listUsers and plain updateVirtualMachine with id works, but as soon as I add the extraconfig parameter, it keeps complaining it is not correct. This is so far with GET request only, I have not yet tried POST, as I haven't taken a look if I should do every value with post (command, id, extraconfig, response and signature?) or just the extraconfig and how signing would go then.

I have tried to sign the request with: "command=updatevirtualmachine&id=xxxx&extraconfig=urlencodedconfig"
both fully lowercase, as well lowercase only the command.

I also tried without adding extraconfig to the signature generation but they all end up with:

    ["errorcode"]=>
    int(401)
    ["errortext"]=>
    string(58) "unable to verify user credentials and/or request signature"

Currently using PHP to do the signatures and it looks like as following:

<?php
$baseurl = "http://localhost:8080/client/api?";

$response = "response=json";

$vmid = "id=6f695b14-71b4-4deb-bc4c-851580324f91";

$apikey = "apikey=<apikey>";
$secretkey = "<secret>";
$configxml = <<<EOD
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </rng>
EOD;

$extraconfig = "extraconfig=" . urlencode($configxml);

$command = "command=updateVirtualMachine&".$vmid;

$hash = hash_hmac("sha1",strtolower($apikey . "&" . $command) ."&" . $extraconfig .  "&" . $response,$secretkey, true);
$base64encoded = base64_encode($hash);
$signature = "signature=" . urlencode($base64encoded);

$link = $baseurl  . $apikey . "&" . $command . "&" . $extraconfig . "&" .$response . "&" . $signature;

$cURLConnection = curl_init();

curl_setopt($cURLConnection, CURLOPT_URL, $link);
curl_setopt($cURLConnection, CURLOPT_RETURNTRANSFER, true);

$json_output = curl_exec($cURLConnection);
curl_close($cURLConnection);

$jsonArrayResponse = json_decode($json_output,true);

var_dump($jsonArrayResponse);


The URL end up looking like this:

http://localhost:8080/client/api?apikey=<Apikey>&command=updateVirtualMachine&id=<vmid>&extraconfig=%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x0%27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x0%27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Cmemballoon+model%3D%27virtio%27%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x04%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A++++%3C%2Fmemballoon%3E%0A++++%3Crng+model%3D%27virtio%27%3E%0A++++++%3Cbackend+model%3D%27random%27%3E%2Fdev%2Furandom%3C%2Fbackend%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x05%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A++++%3C%2Frng%3E&response=json&signature=<signature>

Thanks,

JeanPaul
________________________________
From: JeanPaul van der Mijle <jp...@hyperai.ai>
Sent: Sunday, February 12, 2023 6:41 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

Hi Wei,

Thanks, going to take a deeper look into this.

I assume I will have to add the list as following, to allow the XML below to be accepted by the API when I submit this encoded?

I set this to be allowed for KVM: domain,devices,hostdev,source,address,memballoon,rng,backend

It should reflect this XML part:

<domain>
  <device>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </rng>
  </device>
</domain>

I only need to figure out if domain and device are required or not but I will see when I test it out.

Thanks so far!

JeanPaul
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Sunday, February 12, 2023 5:23 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

I think you need to add additional configuration to vm.
Refer to
https://www.shapeblue.com/cloudstack-feature-first-look-enable-sending-of-arbitrary-configuration-data-to-vms/

-Wei



On Sunday, 12 February 2023, JeanPaul van der Mijle <
jp.vandermijle@hyperai.ai> wrote:

> Hi all,
>
> Does anyone have a good guide on adding a PCI-E card (A Mellanox Connect-X
> 5 VF interface) to cloudstack?
> I had this all running before in stock Qemu, and all settings are still
> present for the card and VF's, I just need to know how the management sees
> that I have additional cards that I have available to assign to VMs.
>
> I tried adding:
> pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
>
> However, after restarting the agent, I noticed that it added escape
> characters:
> pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|100GE,c1\:01.1
>
> Not sure if this is fine, but so far it was the only thing I could find
> back on google, but this is 10 years ago. I suspect this is no longer
> sufficient.
> After restarting the agent, I do not see any changes unfortunately.
>
> Thanks!
>
> JeanPaul
>
>

Re: PCI-E Passthrough

Posted by JeanPaul van der Mijle <jp...@hyperai.ai>.
Hi Wei, All,

I am currently stuck with the API call, I manage to sign requests as listUsers and plain updateVirtualMachine with id works, but as soon as I add the extraconfig parameter, it keeps complaining it is not correct. This is so far with GET request only, I have not yet tried POST, as I haven't taken a look if I should do every value with post (command, id, extraconfig, response and signature?) or just the extraconfig and how signing would go then.

I have tried to sign the request with: "command=updatevirtualmachine&id=xxxx&extraconfig=urlencodedconfig"
both fully lowercase, as well lowercase only the command.

I also tried without adding extraconfig to the signature generation but they all end up with:

    ["errorcode"]=>
    int(401)
    ["errortext"]=>
    string(58) "unable to verify user credentials and/or request signature"

Currently using PHP to do the signatures and it looks like as following:

<?php
$baseurl = "http://localhost:8080/client/api?";

$response = "response=json";

$vmid = "id=6f695b14-71b4-4deb-bc4c-851580324f91";

$apikey = "apikey=<apikey>";
$secretkey = "<secret>";
$configxml = <<<EOD
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </rng>
EOD;

$extraconfig = "extraconfig=" . urlencode($configxml);

$command = "command=updateVirtualMachine&".$vmid;

$hash = hash_hmac("sha1",strtolower($apikey . "&" . $command) ."&" . $extraconfig .  "&" . $response,$secretkey, true);
$base64encoded = base64_encode($hash);
$signature = "signature=" . urlencode($base64encoded);

$link = $baseurl  . $apikey . "&" . $command . "&" . $extraconfig . "&" .$response . "&" . $signature;

$cURLConnection = curl_init();

curl_setopt($cURLConnection, CURLOPT_URL, $link);
curl_setopt($cURLConnection, CURLOPT_RETURNTRANSFER, true);

$json_output = curl_exec($cURLConnection);
curl_close($cURLConnection);

$jsonArrayResponse = json_decode($json_output,true);

var_dump($jsonArrayResponse);


The URL end up looking like this:

http://localhost:8080/client/api?apikey=<Apikey>&command=updateVirtualMachine&id=<vmid>&extraconfig=%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x0%27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x0%27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Cmemballoon+model%3D%27virtio%27%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x04%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A++++%3C%2Fmemballoon%3E%0A++++%3Crng+model%3D%27virtio%27%3E%0A++++++%3Cbackend+model%3D%27random%27%3E%2Fdev%2Furandom%3C%2Fbackend%3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%270x05%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A++++%3C%2Frng%3E&response=json&signature=<signature>

Thanks,

JeanPaul
________________________________
From: JeanPaul van der Mijle <jp...@hyperai.ai>
Sent: Sunday, February 12, 2023 6:41 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

Hi Wei,

Thanks, going to take a deeper look into this.

I assume I will have to add the list as following, to allow the XML below to be accepted by the API when I submit this encoded?

I set this to be allowed for KVM: domain,devices,hostdev,source,address,memballoon,rng,backend

It should reflect this XML part:

<domain>
  <device>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </rng>
  </device>
</domain>

I only need to figure out if domain and device are required or not but I will see when I test it out.

Thanks so far!

JeanPaul
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Sunday, February 12, 2023 5:23 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

I think you need to add additional configuration to vm.
Refer to
https://www.shapeblue.com/cloudstack-feature-first-look-enable-sending-of-arbitrary-configuration-data-to-vms/

-Wei



On Sunday, 12 February 2023, JeanPaul van der Mijle <
jp.vandermijle@hyperai.ai> wrote:

> Hi all,
>
> Does anyone have a good guide on adding a PCI-E card (A Mellanox Connect-X
> 5 VF interface) to cloudstack?
> I had this all running before in stock Qemu, and all settings are still
> present for the card and VF's, I just need to know how the management sees
> that I have additional cards that I have available to assign to VMs.
>
> I tried adding:
> pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
>
> However, after restarting the agent, I noticed that it added escape
> characters:
> pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|100GE,c1\:01.1
>
> Not sure if this is fine, but so far it was the only thing I could find
> back on google, but this is 10 years ago. I suspect this is no longer
> sufficient.
> After restarting the agent, I do not see any changes unfortunately.
>
> Thanks!
>
> JeanPaul
>
>

Re: PCI-E Passthrough

Posted by JeanPaul van der Mijle <jp...@hyperai.ai>.
Hi Wei,

Thanks, going to take a deeper look into this.

I assume I will have to add the list as following, to allow the XML below to be accepted by the API when I submit this encoded?

I set this to be allowed for KVM: domain,devices,hostdev,source,address,memballoon,rng,backend

It should reflect this XML part:

<domain>
  <device>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </rng>
  </device>
</domain>

I only need to figure out if domain and device are required or not but I will see when I test it out.

Thanks so far!

JeanPaul
________________________________
From: Wei ZHOU <us...@gmail.com>
Sent: Sunday, February 12, 2023 5:23 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough

I think you need to add additional configuration to vm.
Refer to
https://www.shapeblue.com/cloudstack-feature-first-look-enable-sending-of-arbitrary-configuration-data-to-vms/

-Wei



On Sunday, 12 February 2023, JeanPaul van der Mijle <
jp.vandermijle@hyperai.ai> wrote:

> Hi all,
>
> Does anyone have a good guide on adding a PCI-E card (A Mellanox Connect-X
> 5 VF interface) to cloudstack?
> I had this all running before in stock Qemu, and all settings are still
> present for the card and VF's, I just need to know how the management sees
> that I have additional cards that I have available to assign to VMs.
>
> I tried adding:
> pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
>
> However, after restarting the agent, I noticed that it added escape
> characters:
> pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|100GE,c1\:01.1
>
> Not sure if this is fine, but so far it was the only thing I could find
> back on google, but this is 10 years ago. I suspect this is no longer
> sufficient.
> After restarting the agent, I do not see any changes unfortunately.
>
> Thanks!
>
> JeanPaul
>
>

Re: PCI-E Passthrough

Posted by Wei ZHOU <us...@gmail.com>.
I think you need to add additional configuration to vm.
Refer to
https://www.shapeblue.com/cloudstack-feature-first-look-enable-sending-of-arbitrary-configuration-data-to-vms/

-Wei



On Sunday, 12 February 2023, JeanPaul van der Mijle <
jp.vandermijle@hyperai.ai> wrote:

> Hi all,
>
> Does anyone have a good guide on adding a PCI-E card (A Mellanox Connect-X
> 5 VF interface) to cloudstack?
> I had this all running before in stock Qemu, and all settings are still
> present for the card and VF's, I just need to know how the management sees
> that I have additional cards that I have available to assign to VMs.
>
> I tried adding:
> pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
>
> However, after restarting the agent, I noticed that it added escape
> characters:
> pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|100GE,c1\:01.1
>
> Not sure if this is fine, but so far it was the only thing I could find
> back on google, but this is 10 years ago. I suspect this is no longer
> sufficient.
> After restarting the agent, I do not see any changes unfortunately.
>
> Thanks!
>
> JeanPaul
>
>