You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "bhouse-nexthop (via GitHub)" <gi...@apache.org> on 2025/12/18 22:52:50 UTC
[D] Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor [cloudstack]
GitHub user bhouse-nexthop created a discussion: Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor
I am running Cloudstack 4.22.0.0 + PR #11932. I have these configurations set:
* mem.overprovisioning.factor: 4
* cluster.memory.allocated.capacity.disablethreshold: 0.85
* vm.min.memory.equals.memory.divided.by.mem.overprovisioning.factor: false
Allocated capacity looks like this:
<img width="367" height="123" alt="image" src="https://github.com/user-attachments/assets/3b945534-9f95-4c2e-a40f-2ed41f607506" />
But actual memory usage looks like this:
<img width="778" height="231" alt="image" src="https://github.com/user-attachments/assets/140aee51-ac50-4169-b5ef-9ed19cc7f841" />
This proves the PR #11932 + the fact that I tuned KSM on the host is working nicely.
That said, with cluster.memory.allocated.capacity.disablethreshold at 85%, I get:
```
2025-12-18 22:35:07,143 WARN [c.c.d.FirstFitPlanner] (API-Job-Executor-67:[ctx-c19a2c63, job-23855, ctx-d91b5367]) (logid:8ecbd546) Cannot allocate cluster list [1] for VM creation since their allocated percentage crosses the disable capacity threshold defined at each cluster at Global Settings Configuration [name: cluster.memory.allocated.capacity.disablethreshold, value: 0.85] for capacity Type : MEMORY, skipping these clusters
```
I tried setting the value to 0 thinking that means disable (not sure why you'd ever use a value of 0 as that would make it always fail), but it didn't fix the error.
I currently have it set to 1 for 100% but I have a feeling in a few days I'm going to hit the same error again even though I haven't hit my overprovision ratio.
Either this check needs to have the ability to be completely disabled, or it needs to be made to honor the overprovisioning factor.
Anyone have any guidance on this?
GitHub link: https://github.com/apache/cloudstack/discussions/12301
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor
thanks @bhouse-nexthop , if you care to add a documentation PR?!
GitHub link: https://github.com/apache/cloudstack/discussions/12301#discussioncomment-15300582
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor [cloudstack]
Posted by "rajujith (via GitHub)" <gi...@apache.org>.
GitHub user rajujith added a comment to the discussion: Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor
Its already documnted.
_Only Instances deployed after the change are affected by the new setting. If you want Instances deployed before the change to adopt the new over-provisioning factor, you must stop and restart the Instances. When this is done, CloudStack recalculates or scales the used and reserved capacities based on the new over-provisioning factors, to ensure that CloudStack is correctly tracking the amount of free capacity._
https://docs.cloudstack.apache.org/en/4.22.0.0/adminguide/hosts.html#setting-over-provisioning-factors
GitHub link: https://github.com/apache/cloudstack/discussions/12301#discussioncomment-15314318
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor [cloudstack]
Posted by "bradh352 (via GitHub)" <gi...@apache.org>.
GitHub user bradh352 added a comment to the discussion: Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor
This is NOT the actual behavior though. Stopping a VM and starting it again does NOT recalculate. Only newly created instances are impacted. The DB edit route was the only successful way I could do this.
GitHub link: https://github.com/apache/cloudstack/discussions/12301#discussioncomment-15318250
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
GitHub user bhouse-nexthop added a comment to the discussion: Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor
I'm monitoring actual memory usage per hypervisor separately. These are for developer VMs so we know they spike and release. It is understood that any sort of overcommit could lead to significant issues if not properly monitored.
It appears setting the `mem.overprovisioning.factor` only applies to VMs *created* newly. It doesn't appear there is any way to change this setting. Even stopping and starting VMs doesn't change it, we can see the overprovisioning factor set in the VM settings stay at whatever the original value was.
I had to make a direct DB edit:
```
update vm_instance_details set value=4 where name='memoryOvercommitRatio';
```
So I guess this is expected behavior, but doesn't appear to be obvious. Its not entirely clear why it works this way.
GitHub link: https://github.com/apache/cloudstack/discussions/12301#discussioncomment-15298683
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor [cloudstack]
Posted by "bhouse-nexthop (via GitHub)" <gi...@apache.org>.
GitHub user bhouse-nexthop closed a discussion: Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor
I am running Cloudstack 4.22.0.0 + PR #11932. I have these configurations set:
* mem.overprovisioning.factor: 4
* cluster.memory.allocated.capacity.disablethreshold: 0.85
* vm.min.memory.equals.memory.divided.by.mem.overprovisioning.factor: false
Allocated capacity looks like this:
<img width="367" height="123" alt="image" src="https://github.com/user-attachments/assets/3b945534-9f95-4c2e-a40f-2ed41f607506" />
But actual memory usage looks like this:
<img width="778" height="231" alt="image" src="https://github.com/user-attachments/assets/140aee51-ac50-4169-b5ef-9ed19cc7f841" />
This proves the PR #11932 + the fact that I tuned KSM on the host is working nicely.
That said, with cluster.memory.allocated.capacity.disablethreshold at 85%, I get:
```
2025-12-18 22:35:07,143 WARN [c.c.d.FirstFitPlanner] (API-Job-Executor-67:[ctx-c19a2c63, job-23855, ctx-d91b5367]) (logid:8ecbd546) Cannot allocate cluster list [1] for VM creation since their allocated percentage crosses the disable capacity threshold defined at each cluster at Global Settings Configuration [name: cluster.memory.allocated.capacity.disablethreshold, value: 0.85] for capacity Type : MEMORY, skipping these clusters
```
I tried setting the value to 0 thinking that means disable (not sure why you'd ever use a value of 0 as that would make it always fail), but it didn't fix the error.
I currently have it set to 1 for 100% but I have a feeling in a few days I'm going to hit the same error again even though I haven't hit my overprovision ratio.
Either this check needs to have the ability to be completely disabled, or it needs to be made to honor the overprovisioning factor.
Anyone have any guidance on this?
GitHub link: https://github.com/apache/cloudstack/discussions/12301
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor [cloudstack]
Posted by "rajujith (via GitHub)" <gi...@apache.org>.
GitHub user rajujith edited a comment on the discussion: Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor
Its already documented.
_Only Instances deployed after the change are affected by the new setting. If you want Instances deployed before the change to adopt the new over-provisioning factor, you must stop and restart the Instances. When this is done, CloudStack recalculates or scales the used and reserved capacities based on the new over-provisioning factors, to ensure that CloudStack is correctly tracking the amount of free capacity._
https://docs.cloudstack.apache.org/en/4.22.0.0/adminguide/hosts.html#setting-over-provisioning-factors
GitHub link: https://github.com/apache/cloudstack/discussions/12301#discussioncomment-15314318
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Disable cluster.memory.allocated.capacity.disablethreshold check? Bad interaction with mem.overprovisioning.factor
you can increase the mem.overprovisioning.factor, or set threshold value to a number larger than 1 by cmk （not sure it affects anything）
however, you might face OOM errors, so please be very careful.
GitHub link: https://github.com/apache/cloudstack/discussions/12301#discussioncomment-15296836
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org