You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "bradh352 (via GitHub)" <gi...@apache.org> on 2025/10/08 13:43:53 UTC
[D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
GitHub user bradh352 created a discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
Since it is not possible to currently pass multiple VLANs through to a VM via a single interface, and I have the need to forward a few hundred(!) VLANs through to a guest, I'm trying to explore my options. It appears at around 23 vNICs you start getting failures, so adding a few hundred vNICs to the guest isn't an option. From what I've read this is due to virtio-net consuming a PCI slot per vNIC and there are only 32 slots available total.
I think the solution is going to be to use Network VF / SR-IOV / PCI-passthrough. From what I can tell, that would mean using "extraconfig" in the VM to tell it to pass through the PCIe VF like:
https://gist.github.com/rajujith/4cc3f17379b63e86f73b041f2be75528#file-enable-gpu-passthrough-on-cloudstack-kvm-with-extraconfig
I can live with that. But how do I pin this VM to this host? I have "DRS" enabled, so it *might* try to migrate it, wouldn't it? If there isn't a way to pin the VM is there a way to exclude from DRS?
Also, what about maintenance mode? That tries to live migrate VMs away, but in this case I'd need to shut it down instead. I see something about rolling maintenance scripts/hooks:
https://www.shapeblue.com/cloudstack-feature-first-look-kvm-rolling-maintenance/
We've up to now just used the normal maintenance mode before doing tasks on a host. But we deploy with Ansible so maybe we just add the shutdown/startup there ... but it would be convenient to have a hook if someone needs to power a host down for another reason such as changing out a memory stick or similar. Will those same hooks be called?
Anyhow, am I on the right path here for my needs? Any insight would be helpful!
===
Just because I know someone is going to ask why I would possibly need this, the scenario is I have some specialized equipment that must be fully isolated from eachother (of which we have a few hundred) so can't reside in the same vlan. They also need L2 adjacency for all services because their default gateways get reconfigured for special needs in their environments. And finally, we need to be able to access this equipment but appear to be L2 adjacent so we SNAT out that port so all traffic appears to originate from their L2 segment from them. Yeah, its bad. No, there's no way to "fix" it.
GitHub link: https://github.com/apache/cloudstack/discussions/11808
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "vishesh92 (via GitHub)" <gi...@apache.org>.
GitHub user vishesh92 added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
@bhouse-nexthop @bradh352 I missed this one. As @erikbocks said, you can passthrough any types of pci device as long as it shows up in the lspci output and supports passthrough with KVM. You just need to make the change like https://github.com/apache/cloudstack/pull/11715 .
For it to show network VFs, that might require some extra changes in the script to discover the VFs for a network card. `scripts/vm/hypervisor/kvm/gpudiscovery.sh` is available at `/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/gpudiscovery.sh` on a KVM host. You can make this change on a single host and discover GPU devices on the host. This will make the PCI device visible only on that host only. After this, you will need to create a compute offering with the discovered PCI device.
You can try updating it and see if it works. This shouldn't require a new release for you to test it out. Let me know how this goes.
P.S. - I know some people in the community tested passthrough of NPUs, GPUs, processing accelerators & VGA Controllers with CloudStack. I haven't tested with network cards and your use case specifically.
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-16515618
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
> I am working on the final touches of the spec before starting the development. As soon as it is ready, I will post them here for you guys to gather some feedback. I plan to propose/present it to the community in january/2026, but it could be ready before that.
@erikbocks , anything you can link us to for evaluation/review? (if you are still planning this)
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-16515674
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
GitHub user bhouse-nexthop added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
awesome!
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-15182517
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "erikbocks (via GitHub)" <gi...@apache.org>.
GitHub user erikbocks added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
@DaanHoogland I am still working in the specification. As soon as I have it finished, I will post it for review.
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-16546217
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "erikbocks (via GitHub)" <gi...@apache.org>.
GitHub user erikbocks added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
Hello, @bhouse-nexthop and @DaanHoogland,
I am currently working on a specification and development effort that will provide native support for the PCI passthrough management method in Apache CloudStack. It should work similarly to the feature introduced in #11143, but encompass all types of PCI devices, including GPUs, network cards, and block devices.
The feature aims to allow operators to provide/define compute offerings that enable users to consume/allocate PCI devices in a simple way. Besides that, the feature will also handle the automatic discovery of host's devices and any physical changes, decreasing the amount of manual labor currently needed for operators, and turning the process less error-prone and time-consuming.
I am working on the final touches of the spec before starting the development. As soon as it is ready, I will post them here for you guys to gather some feedback. I plan to propose/present it to the community in january/2026, but it could be ready before that.
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-15177769
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
GitHub user bhouse-nexthop added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
so the workaround "worked" to use multiple host tags and effectively pin the vm.
That said, isn't GPU passthrough that was recently added using basically the same thing as NIC passthrough? Any reason it wasn't made more generic as just "Device" passthrough?
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-15173706
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
GitHub user bhouse-nexthop added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
I thought about vxlan as well, but without hardware offload within a VM, I was worried it could be a bottleneck. Also, interestingly, while we're using vxlan-evpn, our switches aren't actually participating in it at this time due to some legacy constraints.
Thanks for the hint on pinning using host tags. Can a host have more than 1 tag? I honestly haven't tried.
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-14639052
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "StepBee (via GitHub)" <gi...@apache.org>.
GitHub user StepBee added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
Yes, you can give each host multiple tags, enabling a variety of functionality when combining them.
In regards to the VXLAN - depending on how much traffic you are talking about, but if you have the possibility i think it's always worth a try.
I imagine finding a way out of the pure Layer 2 dependency could help for other challenges as well.
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-14640057
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
@bradh352 @StepBee , is there a conclusion to be made here? issues with feature requests to be opened? documentation to be added?
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-15135543
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest) [cloudstack]
Posted by "StepBee (via GitHub)" <gi...@apache.org>.
GitHub user StepBee added a comment to the discussion: Network VF (SR-IOV) options / questions (need a few hundred VLANs in guest)
Hi @bradh352 ,
an interesting case you have, and i see the requirement for VLAN-trunk enabled VMs coming up in more and more requests - one of them being [https://github.com/apache/cloudstack/issues/11491](https://github.com/apache/cloudstack/issues/11491) where you are involved already as well.
In your specific usecase an idea which came up to my mind is the use of VXLAN.
Maybe you could encapsulate your VLANs into VNIs and setup a VXLAN connection between your networking devices (switches?) and your VM within cloudstack via a layer 3 path - so your VM being a VTEP for the VXLAN connection as well?
I didn't fully think it through but your scenario sounds special and special requirements sometimes require special solutions.
In regards to pinning a VM to a host - one very simple solution would be the use of a specific host-tag (i.e. sriovnic01_host01) and creating a compute offering using this host-tag. As a result the VM will always we scheduled to run on the host with this specific tag.
Best regards,
Stephan
GitHub link: https://github.com/apache/cloudstack/discussions/11808#discussioncomment-14635580
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org