You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "bernardodemarco (via GitHub)" <gi...@apache.org> on 2026/03/27 22:34:58 UTC
[I] Live scaling for VMs with fixed service offerings on KVM [cloudstack]
bernardodemarco opened a new issue, #12908:
URL: https://github.com/apache/cloudstack/issues/12908
# Live scaling for VMs with fixed service offerings on KVM
This specification introduces a new feature enabling users to live scale VMs created with fixed service offerings on the KVM hypervisor.
## Table of contents
- [Live scaling for VMs with fixed service offerings on KVM](#live-scaling-for-vms-with-fixed-service-offerings-on-kvm)
- [Table of contents](#table-of-contents)
- [1. Problem description](#1-problem-description)
- [2. Proposed changes](#2-proposed-changes)
- [2.1. Apache CloudStack definition of domain XMLs with KVM](#21-apache-cloudstack-definition-of-domain-xmls-with-kvm)
- [2.2. New cluster-wide settings to control the memory and vCPUs maximum capacity for live scaling](#22-new-cluster-wide-settings-to-control-the-memory-and-vcpus-maximum-capacity-for-live-scaling)
- [2.3. `scaleVirtualMachine` API](#23-scalevirtualmachine-api)
- [3.0. Conclusion and Limitations](#30-conclusion-and-limitations)
## 1. Problem description
Currently, Apache CloudStack supports three types of compute offerings: fixed, custom constrained and custom unconstrained offerings. Fixed offerings have a fixed number of vCPUs, vCPU speed, and RAM memory. Custom constrained offerings accept a fixed vCPU speed, and a range of vCPUs and RAM memory in which users can select a value from. Lastly, custom unconstrained offerings accept an arbitrary amount of vCPUs, vCPU speed and RAM memory.
When using KVM as hypervisor, Apache CloudStack supports scaling `Stopped` instances by executing the `scaleVirtualMachine` API. During this process, since the VMs are stopped, their metadata is updated in the database and, thus, when the VMs are later started again, their respective domain XMLs are created with the updated attributes.
For running VMs, Apache CloudStack only supports live scaling VMs with custom constrained and custom unconstrained compute offerings[^enable-scale-vm-setting]. For VMs with fixed compute offerings, users necessarily need to stop the VMs, change their service offerings through the `scaleVirtualMachine` API and start them again. However, depending on the criticality of the applications running on the VMs, the downtime caused by the scaling process is highly undesirable.
## 2. Proposed changes
To address the described problem, this specification introduces the feature of live scaling VMs with fixed service offerings when using the KVM hypervisor on Apache CloudStack cloud environments. A high-level design of the feature is presented, briefly describing the proposed changes to the generation of guest VM domain XMLs, the `scaleVirtualMachine` API workflows and the global settings controlling the scaling process.
### 2.1. Apache CloudStack definition of domain XMLs with KVM
During the process of deploying virtual machines with KVM, a VM transfer object (`VirtualMachineTO`) is implemented given a `VirtualMachineProfile`, which is an object that stores the attributes of the VM. The implementation of VMs transfer objects is illustrated in the following activity diagram:
<img width="361" height="921" alt="Image" src="https://github.com/user-attachments/assets/f6dce360-5f80-4c6f-ab8c-ab8bc634a2e9" />
When configuring the memory and vCPU attributes for the VMs, Apache CloudStack implements the following workflow:
<img width="849" height="657" alt="Image" src="https://github.com/user-attachments/assets/aad3adc2-758a-417a-aa73-34e97d8c2827" />
Regarding the definition of domain XMLs for guest VMs, the following boilerplate is used by the Apache CloudStack Agent:
```xml
<domain type='kvm' id='4'>
<!--(...)-->
<maxMemory slots='16' unit='KiB'>1930240</maxMemory>
<memory unit='KiB'>1048576</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu placement='static' current='1'>2</vcpu>
<!--(...)-->
<cpu mode='custom' match='exact' check='full'>
<!--(...)-->
<numa>
<cell id='0' cpus='0-1' memory='1048576' unit='KiB'/>
</numa>
<!--(...)-->
</cpu>
</domain>
```
Regarding vCPU configuration, the content of the `vcpu` element defines the maximum number of vCPUs that can be allocated for the guest VM. Its `current` property defines the number of vCPUs that are effectively active[^cpu-configuration-libvirt-docs]. As can be analyzed from the above activity diagrams, for custom constrained compute offerings, the content of the `vcpu` element is defined as the offering's maximum number of vCPUs. For custom unconstrained offerings, the element's content is defined as the `vm.serviceoffering.cpu.cores.max` global setting value, defaulting to the host maximum CPU capacity in scenarios that the global setting is equal to zero.
As for guest VMs memory configuration, the `maxMemory` element defines the maximum memory allocation for the VM. The `memory` element represents the maximum amount of memory available for the VM at boot time. The `currentMemory` represents the actual allocation of memory for the VM[^memory-configuration-libvirt-docs]. Lastly, Libvirt currently requires the specification of NUMA nodes to enable memory hotplug. Each `cell` element represents a NUMA node. The `cpus` attribute define the range of CPUs that are part of the node. The `memory` attribute represents the amount of memory in use by the VM[^numa-configuration-libvirt-docs].
As can be noticed from the VM configuration workflow, for custom constrained compute offerings, the `maxMemory` element content is defined as the offering's maximum number of memory. For custom unconstrained offerings, the element's content is defined as the `vm.serviceoffering.ram.size.max` global setting value, defaulting to the host maximum memory capacity in scenarios in which the global setting is equal to zero.
Hence, to address the live scaling of VMs with fixed offerings, the first validation on the configuration of memory and vCPU for a guest VM, that checks whether the VM is dynamically scalable, will be modified. Currently, it considers a VM to be dynamically scalable if its offering is dynamic (that is, either the amount of vCPUs, vCPU speed or RAM memory is not specified); the VM's `dynamically_scalable` property is `true`; and the global setting `enable.dynamic.scale.vm` is true.
Therefore, the check for dynamic offerings will be removed from the above mentioned validation. As a consequence of that, when the `dynamically_scalable` property of VMs is `true` and the `enable.dynamic.scale.vm` global setting is `true`, then the domain XMLs of guest VMs will always have a range of memory and vCPUs to be scaled up to, even when the VMs are created from fixed compute offerings.
To define the upper limit of the memory and vCPU live scaling range, the `kvm.cpu.dynamic.scaling.capacity` and `kvm.memory.dynamic.scaling.capacity` global settings will be considered (see [new cluster-wide settings](#22-new-cluster-wide-settings-to-control-the-memory-and-vcpus-maximum-capacity-for-live-scaling)). Therefore, the maximum number of vCPUs will be retrieved from the `kvm.cpu.dynamic.scaling.capacity` setting value and the maximum number of memory will be retrieved from the `kvm.memory.dynamic.scaling.capacity` value. When these settings are equal to zero, the host maximum capacity of CPU and memory will be considered.
This procedure will also be applied for custom constrained offerings to enable seamless live scaling between all types of compute offerings existing in Apache CloudStack. Thus, at the KVM level, all types of VMs hosted in a given host will be homogeneous regarding the CPU and memory upper limits. At the Apache CloudStack level, on the other hand, the Management Server will be responsible for validating the computing resources ranges defined in the constrained offerings.
### 2.2. New cluster-wide settings to control the memory and vCPUs maximum capacity for live scaling
As mentioned on the [domain XMLs definition section](#21-apache-cloudstack-definition-of-domain-xmls-with-kvm), the `vm.serviceoffering.cpu.cores.max` and `vm.serviceoffering.ram.size.max` global settings are used to control de maximum amount of vCPUs and memory to which VMs with custom unconstrained offerings can be live scaled to. However, both settings are also currently used to limit the maximum amount of CPU and RAM that can be defined for compute offerings and allocated to VMs.
To segregate goals and responsibilities, two new cluster-wide settings will be introduced:
| Name | Type | Scope | Description |
| ------ | ------ | ------ | ----------- |
| `kvm.memory.dynamic.scaling.capacity` | Integer | Cluster | Defines the maximum memory capacity in MiB for which VMs can be dynamically scaled to with KVM. The `kvm.memory.dynamic.scaling.capacity` setting's value will be used to define the value of the `<maxMemory />` element of domain XMLs. If it is set to a value less than or equal to `0`, then the host's memory capacity will be considered. |
| `kvm.cpu.dynamic.scaling.capacity` | Integer | Cluster | Defines the maximum vCPU capacity for which VMs can be dynamically scaled to with KVM. The `kvm.cpu.dynamic.scaling.capacity` setting's value will be used to define the value of the `<vcpu />` element of domain XMLs. If it is set to a value less than or equal to `0`, then the host's CPU cores capacity will be considered. |
Therefore, both settings will be used to exclusively control the maximum live scaling capacity for memory and vCPUs. To maintain compatibility with the current behavior, the values of the `vm.serviceoffering.cpu.cores.max` and `vm.serviceoffering.ram.size.max` global settings will be used to populate the initial values of the new cluster-wide settings.
### 2.3. `scaleVirtualMachine` API
The following activity diagram illustrates the current implementation of the VM scaling process:
<img width="951" height="982" alt="Image" src="https://github.com/user-attachments/assets/13c8e18c-9ec3-4f42-b334-d4f9cf3331a1" />
As can be noticed, when upgrading running VMs, after performing some general validations, the Management Server checks if the VM is running on KVM, and if its offering is not dynamic (that is, vCPU, vCPU speed and RAM memory are specified for the offering). If the VM meets these conditions, then an exception is thrown, informing the end user that KVM does not support live scaling VMs with fixed compute offerings.
Therefore, the above-mentioned check for dynamic offerings will be removed from the scaling workflow. This is possible because the domain XMLs of guest VMs will be prepared to support live scaling when the global setting `enable.dynamic.scale.vm` and the VM's `dynamically_scalable` property are set to `true`.
Additionally, it is relevant to note that the current workflow does not update the VM's CPU quota percentage based on its new CPU frequency. To address this, the current implementation will be extended so that, before building the `ScaleVmCommand`, the new CPU quota percentage is calculated and included in the command payload, along with a flag indicating whether the CPU cap has changed between the old and new service offerings.
At the Agent side, the following workflow will be executed:
<img width="1059" height="651" alt="Image" src="https://github.com/user-attachments/assets/1b3c67c1-002b-4edc-9d32-a1ec38478a22" />
Thus, if both the old and the new service offerings have CPU limitation enabled, the domain's CPU quota will be updated on the fly. If the old service offering does not have CPU limitation enabled but the new one does, the domain's CPU quota and period parameters will be set. Conversely, it the old offering has CPU limitation enabled and the new one does not, these parameters will be removed from the domain.
To update the CPU scheduling parameters of a domain on the fly, the `virDomainSetSchedulerParameters` Libvirt API will be used. To remove CPU limitation from a domain, the API will be called with the `vcpu_quota` parameter set to `17,592,186,044,415`. Due to a Libvirt regression (see [Libvirt regression description](https://gitlab.com/libvirt/libvirt/-/work_items/324)), it is not possible to set this value to `-1`. However, setting it to `17,592,186,044,415` has the same effect, ensuring that CPU limitations are effectively removed from running domains despite the Libvirt's constraint.
# 3.0. Conclusion and Limitations
This proposal introduces support for live scaling of VMs with fixed service offerings when using the KVM hypervisor in Apache CloudStack environments. As a limitation, it is important to highlight that live scaling will not be supported for existing domains created prior to upgrading to the Apache CloudStack release in which this patch is included. This is because their domain XMLs are not prepared to support live CPU and memory scaling.
To enable live scaling for such VMs, the following steps are required:
1. Ensure that the `enable.dynamic.scale.vm` global setting is enabled.
2. Ensure that the VM's template is marked as dynamically scalable.
3. Ensure that the VM itself is marked as dynamically scalable.
[^enable-scale-vm-setting]: The global setting `enable.dynamic.scale.vm` controls whether the live scaling feature is enabled or not.
[^cpu-configuration-libvirt-docs]: More information about the configuration of guest VMs CPU attributes can be found at [the Libvirt documentation](https://libvirt.org/formatdomain.html#cpu-allocation).
[^memory-configuration-libvirt-docs]: More information about the configuration of guest VMs memory attributes can be found at [the Libvirt documentation](https://libvirt.org/formatdomain.html#memory-allocation).
[^numa-configuration-libvirt-docs]: More information about the configuration of guest NUMA topology can be found at [the Libvirt documentation](https://www.libvirt.org/formatdomain.html#cpu-model-and-topology).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Live scaling for VMs with fixed service offerings on KVM [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #12908:
URL: https://github.com/apache/cloudstack/issues/12908#issuecomment-4147760533
@bernardodemarco
proposal looks good overall.
If I understand correctly, this means setting `maxMemory` for VMs with dynamic scaling enabled, right?
I am wondering whether using global configurations is the best approach. In most cases, users tend to scale CPU/memory relative to the current VM size rather than relying on a fixed global limit.
For example:
- Relative scaling (e.g., doubling): 2GB → 4GB, 4GB → 8GB, 8GB → 16GB
- Fixed increment scaling: 2GB → 4GB, 4GB → 6GB, 8GB → 10GB
Would it make more sense to support scaling policies based on the current allocation instead of a global maximum?
The global maximum can remain
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Live scaling for VMs with fixed service offerings on KVM [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on issue #12908:
URL: https://github.com/apache/cloudstack/issues/12908#issuecomment-4157384560
Hello, @weizhouapache
> proposal looks good overall.
Thanks!
> If I understand correctly, this means setting maxMemory for VMs with dynamic scaling enabled, right?
The domain XML of VMs prepared for dynamic scaling includes both `vcpu` element, whose value defines the upper limit for vCPUs, and the `maxMemory` element, whose value defines the upper limit for memory. Both are required by KVM to enable live scaling.
> I am wondering whether using global configurations is the best approach.
Using global configurations is already the approach adopted by ACS. As described in the spec, for custom unconstrained offerings, the value of the `maxMemory` element is derived from the `vm.serviceoffering.ram.size.max` global setting, and the `vcpus` element is derived from the the `vm.serviceoffering.cpu.cores.max` setting. The new settings are introduced to: (i) better separate goals and responsibilities, since the existing settings are also used by unrelated workflows; and (ii) change their scope to be cluster-wide.
> In most cases, users tend to scale CPU/memory relative to the current VM size rather than relying on a fixed global limit.
For end-users, the upper limit for memory and CPU are transparent. The settings `kvm.memory.dynamic.scaling.capacity` and `kvm.cpu.dynamic.scaling.capacity` are intended for operators to define a cluster-wide upper bound for the XML element values. In most cases, these settings will be set to `0` (default value), meaning that the host's maximum CPU and memory capacity will be used to determine the limits. Custom limits are expected to be used only in specific edge scenarios.
> For example:
> Relative scaling (e.g., doubling): 2GB → 4GB, 4GB → 8GB, 8GB → 16GB
> Fixed increment scaling: 2GB → 4GB, 4GB → 6GB, 8GB → 10GB
> Would it make more sense to support scaling policies based on the current allocation instead of a global maximum?
Yes, this makes total sense. However, it is a new concept that ACS currently does not support. The current proposal focuses on leveraging existing features of ACS, i.e., it aims to enable changing the service offering of a VM dynamically, as it is currently already supported for other hypervisors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Live scaling for VMs with fixed service offerings on KVM [cloudstack]
Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel closed issue #12908: Live scaling for VMs with fixed service offerings on KVM
URL: https://github.com/apache/cloudstack/issues/12908
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org