You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "bhouse-nexthop (via GitHub)" <gi...@apache.org> on 2026/09/10 17:01:32 UTC
[PR] Give a system VM one NIC queue per CPU [cloudstack]
bhouse-nexthop opened a new pull request, #14136:
URL: https://github.com/apache/cloudstack/pull/14136
### Description
A system VM gets one NIC queue no matter how many CPUs it has, so every packet interrupt lands on
CPU0. Adding CPUs to a router therefore does not move more packets, which is the only reason to add
them.
The queue count is already a VM detail, `nic.multiqueue.number`, and the agent already turns it into
`<driver queues='N'/>` on the interface. A user VM can set it three ways:
| how | since |
|---|---|
| `deployVirtualMachine nicmultiqueuenumber=N` | 4.18 |
| `updateVirtualMachine details[0].nic.multiqueue.number=N` | |
| the Settings tab - `listDetailOptions` offers the key for KVM | |
A system VM goes through none of them. It is not created by `deployVirtualMachine`, it is not a
`UserVm` so `updateVirtualMachine` does not apply, and it has no details editor. **There is no way
to set it at all**, which is why this is a fix rather than a new setting.
So a system VM now gets one queue per CPU when nothing is set:
| CPUs | emitted |
|---|---|
| 1 | nothing, identical to today |
| 4 | `queues='4'` |
| 512 | `queues='256'` |
The 256 is the tap device ceiling in the host kernel, which refuses the interface rather than
trimming to fit. Above the CPU count it would make no difference anyway - the guest driver uses
`min(CPUs, queues)` and pushes that at probe, so no guest side change is needed.
A queue number already set on the VM still wins.
**The default offering is a single CPU**, so a default install emits the same domain XML as before.
Only a system VM someone has already resized differs, and it differs in the direction the resize
asked for.
### Worth calling out
- **There is no way to turn this off.** There is no way to turn it on today either, so it is
symmetric, but it is worth being explicit about on a release branch.
- **The cost scales with NIC count, not just queue count.** A VPC router has one NIC per tier, so a
4 CPU router fronting 8 tiers goes from 8 vhost threads to 32. Idle queues are cheap, but the
thread count is not nothing on a dense host.
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [x] Minor
- [ ] Trivial
### How Has This Been Tested?
`KVMGuruTest`, `Tests run: 35, Failures: 0`. Six new cases:
- a 4 CPU system VM gets `queues='4'`
- a 512 CPU one stops at the tap ceiling of 256
- a single CPU one is left alone, so no attribute is emitted
- other details on the VM are preserved
- a queue number already on the VM wins
- a user VM is never touched
The guest side was checked against the driver source the system VM runs, Linux 6.1
`drivers/net/virtio_net.c`:
```c
/* Enable multiqueue by default */
if (num_online_cpus() >= max_queue_pairs)
vi->curr_queue_pairs = max_queue_pairs;
else
vi->curr_queue_pairs = num_online_cpus();
```
followed by `virtnet_set_queues(vi, vi->curr_queue_pairs)` at probe, so the queues come up without
`ethtool -L`. The 256 comes from `MAX_TAP_QUEUES` in `drivers/net/tun.c`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-5717825383
> Any idea on an ETA for your implementation?
@bhouse-nexthop, yes, probably no later than the middle or end of next week. I'll ping you guys for review once I submit the PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-5717278413
> Instead of deriving the count unconditionally, could we make it configurable per system/router service offering? A detail on the offering (read into nic.multiqueue.number, still letting an explicit per-VM value win) would keep your CPU-derived default for anyone who sets -1, let an operator pin an explicit N, and leave it off when unset — which also covers the "no way to turn this off" note.
The offering size is pretty small at 1 cpu, which likely most people are sticking with. I know cloudstack likes knobs for everything, and I found out the hard way on a lot of things since there is no documentation on 'recommended knobs to turn on'. I honestly don't see harm in this ... but if its a requirement I can add yet another knob to bit someone else in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-5932052189
@vladimirpetrov @sureshanaparti could you take a look at this one? We'd like this fix to make it into the upcoming 4.22.2 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-6086410967
@bhouse-nexthop, it took more than I was expecting, but here it is the PR #14378 that adds granular and flexible support for multi-queue for VRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "poddm (via GitHub)" <gi...@apache.org>.
poddm commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-5717222136
Instead of deriving the count unconditionally, could we make it configurable per system/router service offering? A detail on the offering (read into nic.multiqueue.number, still letting an explicit per-VM value win) would keep your CPU-derived default for anyone who sets -1, let an operator pin an explicit N, and leave it off when unset — which also covers the "no way to turn this off" note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on code in PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#discussion_r4234279219
##########
server/src/main/java/com/cloud/hypervisor/HypervisorGuruBase.java:
##########
@@ -265,6 +269,34 @@ protected void addServiceOfferingExtraConfiguration(ServiceOffering offering, Vi
}
}
+ /**
+ * Gives a system VM one NIC queue per CPU. The guest only ever uses as many queues as it has
+ * CPUs, so this is what the CPU count already implies, and a system VM is only given more CPUs
+ * to move more packets.
+ *
+ * A user VM sets this per VM with deployVirtualMachine or updateVirtualMachine. A system VM
+ * goes through neither, so there is otherwise no way for it to get a queue at all.
+ *
+ * The default offering is a single CPU, so nothing changes until an operator resizes.
+ */
+ protected void addDefaultNicQueuesForSystemVm(VirtualMachineTO to) {
Review Comment:
> only gap i see is the storage and console proxy vms
Yes. One alternative would be to keep this one for SSVMs and CPVMs and https://github.com/apache/cloudstack/pull/14378 for VRs. I do not know what is the use case to apply multi-queue for CPVMs and SSVMs, but it would be great to have a setting to indicate whether the number of queues should always be equal to the number of vCPUs. This behavior may not be always desired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on code in PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#discussion_r4234006037
##########
server/src/main/java/com/cloud/hypervisor/HypervisorGuruBase.java:
##########
@@ -265,6 +269,34 @@ protected void addServiceOfferingExtraConfiguration(ServiceOffering offering, Vi
}
}
+ /**
+ * Gives a system VM one NIC queue per CPU. The guest only ever uses as many queues as it has
+ * CPUs, so this is what the CPU count already implies, and a system VM is only given more CPUs
+ * to move more packets.
+ *
+ * A user VM sets this per VM with deployVirtualMachine or updateVirtualMachine. A system VM
+ * goes through neither, so there is otherwise no way for it to get a queue at all.
+ *
+ * The default offering is a single CPU, so nothing changes until an operator resizes.
+ */
+ protected void addDefaultNicQueuesForSystemVm(VirtualMachineTO to) {
Review Comment:
The current PR encompasses one specific use case, which is already addressed by #14378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-5717535354
> Hello, @bhouse-nexthop and @poddm
>
> @bhouse-nexthop, thanks for the PR!
>
> I'm finishing the port of a new feature that adds support for granular multi-queue management for the different NIC types of VRs in guest networks and VPCs. Basically, it'll be possible to configure, for VPC offerings:
>
> * The number of queues for the public NIC of VRs belonging to VPCs; and,
> * The number of queues for the private gateway NIC of VRs belonging to VPCs.
>
> And, for network offerings:
>
> * The number of queues for the public NIC of VRs belonging to guest networks; and,
> * The number of queues for the private NIC of VRs belonging to guest networks and VPC tiers.
>
> Similarly to multi-queue support for end-user VMs, if these values are set to `-1`, ACS will automatically configure the number of queues to match the number of vCPUs assigned to the VR.
>
> Therefore, with this approach, we can granularly define the desired number of queues based on the type of network traffic each NIC carries.
ok, that sounds reasonable. Any idea on an ETA for your implementation? In the mean time we're actually running this in production ourselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-6174204923
Closing in favour of #14378, which covers VR multi-queue per offering, including tiers added to a running VPC. Thanks @bernardodemarco and @Damans227.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
bhouse-nexthop closed pull request #14136: Give a system VM one NIC queue per CPU
URL: https://github.com/apache/cloudstack/pull/14136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#discussion_r4234185730
##########
server/src/main/java/com/cloud/hypervisor/HypervisorGuruBase.java:
##########
@@ -265,6 +269,34 @@ protected void addServiceOfferingExtraConfiguration(ServiceOffering offering, Vi
}
}
+ /**
+ * Gives a system VM one NIC queue per CPU. The guest only ever uses as many queues as it has
+ * CPUs, so this is what the CPU count already implies, and a system VM is only given more CPUs
+ * to move more packets.
+ *
+ * A user VM sets this per VM with deployVirtualMachine or updateVirtualMachine. A system VM
+ * goes through neither, so there is otherwise no way for it to get a queue at all.
+ *
+ * The default offering is a single CPU, so nothing changes until an operator resizes.
+ */
+ protected void addDefaultNicQueuesForSystemVm(VirtualMachineTO to) {
Review Comment:
makes sense, 14378 covers the router case including new tiers. only gap i see is the storage and console proxy vms, which 14378 doesnt touch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on code in PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#discussion_r4233990957
##########
server/src/main/java/com/cloud/hypervisor/HypervisorGuruBase.java:
##########
@@ -332,6 +364,8 @@ protected VirtualMachineTO toVirtualMachineTO(VirtualMachineProfile vmProfile) {
addExtraConfig(detailsInVm, to, vm.getAccountId(), vm.getHypervisorType());
}
+ addDefaultNicQueuesForSystemVm(to);
Review Comment:
Based on a quick look at the PR code, it will not work for VPCs VRs. Different workflows are executed when (i) adding new tiers for VPCs and (ii) restarting VPCs with clean-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#discussion_r4233530651
##########
server/src/main/java/com/cloud/hypervisor/HypervisorGuruBase.java:
##########
@@ -332,6 +364,8 @@ protected VirtualMachineTO toVirtualMachineTO(VirtualMachineProfile vmProfile) {
addExtraConfig(detailsInVm, to, vm.getAccountId(), vm.getHypervisorType());
}
+ addDefaultNicQueuesForSystemVm(to);
Review Comment:
what happens when a new tier is added to a running vpc router? looks like that new nic still gets one queue until the router restarts
##########
server/src/main/java/com/cloud/hypervisor/HypervisorGuruBase.java:
##########
@@ -265,6 +269,34 @@ protected void addServiceOfferingExtraConfiguration(ServiceOffering offering, Vi
}
}
+ /**
+ * Gives a system VM one NIC queue per CPU. The guest only ever uses as many queues as it has
+ * CPUs, so this is what the CPU count already implies, and a system VM is only given more CPUs
+ * to move more packets.
+ *
+ * A user VM sets this per VM with deployVirtualMachine or updateVirtualMachine. A system VM
+ * goes through neither, so there is otherwise no way for it to get a queue at all.
+ *
+ * The default offering is a single CPU, so nothing changes until an operator resizes.
+ */
+ protected void addDefaultNicQueuesForSystemVm(VirtualMachineTO to) {
Review Comment:
how does this fit with #14378, which lets admins set the router queue count per offering? if both go in it looks like this one would override theirs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-5717497750
Hello, @bhouse-nexthop and @poddm
@bhouse-nexthop, thanks for the PR!
I'm finishing the port of a new feature that adds support for granular multi-queue management for the different NIC types of VRs in guest networks and VPCs. Basically, it'll be possible to configure, for VPC offerings:
- The number of queues for the public NIC of VRs belonging to VPCs; and,
- The number of queues for the private gateway NIC of VRs belonging to VPCs.
And, for network offerings:
- The number of queues for the public NIC of VRs belonging to guest networks; and,
- The number of queues for the private NIC of VRs belonging to guest networks and VPC tiers.
Similarly to multi-queue support for end-user VMs, if these values are set to `-1`, ACS will automatically configure the number of queues to match the number of vCPUs assigned to the VR.
Therefore, with this approach, we can granularly define the desired number of queues based on the type of network traffic each NIC carries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Give a system VM one NIC queue per CPU [cloudstack]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #14136:
URL: https://github.com/apache/cloudstack/pull/14136#issuecomment-5622506383
## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/14136?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 4.02%. Comparing base ([`d0d49bc`](https://app.codecov.io/gh/apache/cloudstack/commit/d0d49bc864ad2558671552d6f526a1137cba5aea?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7931bc6`](https://app.codecov.io/gh/apache/cloudstack/commit/7931bc6b1e81493d87120bf8b269a15b88fa73dd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on 4.22.
> :exclamation: There is a different number of reports uploaded between BASE (d0d49bc) and HEAD (7931bc6). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (d0d49bc) | HEAD (7931bc6) |
>|------|------|------|
>|unittests|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## 4.22 #14136 +/- ##
=============================================
- Coverage 17.93% 4.02% -13.92%
=============================================
Files 5928 449 -5479
Lines 535174 38239 -496935
Branches 65494 7082 -58412
=============================================
- Hits 95981 1538 -94443
+ Misses 428266 36489 -391777
+ Partials 10927 212 -10715
```
| [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/14136/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/14136/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `4.02% <ø> (ø)` | |
| [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/14136/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/cloudstack/pull/14136?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org