You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "amitmvyas (via GitHub)" <gi...@apache.org> on 2025/11/05 19:34:12 UTC
[I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
amitmvyas opened a new issue, #11994:
URL: https://github.com/apache/cloudstack/issues/11994
Hello,
I have deployed K8s cluster in my CS 4.21.0.0 environment and I am trying to Autoscale the same to add a worker node and I am seeing below message in management server logs.
```
2025-11-05 16:07:17,543 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Scaling Kubernetes cluster KubernetesCluster {"id":71,"name":"CL8S","uuid":"da626491-7457-4106-a165-de0d41d2417a"}
2025-11-05 16:07:17,553 WARN [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!
2025-11-05 16:07:17,553 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Revoking launch permission for systemVM template
2025-11-05 16:07:17,648 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-34:[ctx-228e86cd, job-1437]) (logid:56268dc3) Remove job-1437 from job monitoring
```
I am having a hypevisor as KVM which is running on Ubuntu 24.04.3.
Can anyone help figure out why I am facing this problem and whether KVM is really not supported for CKS Autoscale Feature ? if not, then which hypevisor is supported for Autoscale ?
Regards,
Amit Vyas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3833447160
> I have implemented a fix in KubernetesClusterScaleWorker.java (lines 358-389) to address issues with cluster scaling. Updated code is as follows.
>
Hi @chandrakantrai can you raise PR (against 4.22) with these changes? cc @weizhouapache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3495637915
@weizhouapache that check was for scaling the cluster with a new offering. It shouldn't affect scaling for node count unless something changed recently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "chandrakantrai (via GitHub)" <gi...@apache.org>.
chandrakantrai commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3724012616
I have implemented a fix in KubernetesClusterScaleWorker.java (lines 358-389) to address issues with cluster scaling. Updated code is as follows.
` private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
boolean updateNodeOffering, boolean updateClusterOffering)
throws CloudRuntimeException {
// Do not validate for clusters in Created state (no VMs exist yet to upgrade).
// Validation will be done later only if we are actually upgrading existing VMs.
List<KubernetesCluster.State> scalingStates =
List.of(KubernetesCluster.State.Scaling, KubernetesCluster.State.ScalingStoppedCluster);
if (!scalingStates.contains(kubernetesCluster.getState())) {
stateTransitTo(kubernetesCluster.getId(), KubernetesCluster.Event.ScaleUpRequested);
}
if (KubernetesCluster.State.Created.equals(originalState)) {
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering,
updateNodeOffering, updateClusterOffering);
return;
}
final long size = getNodeCountForType(nodeType, kubernetesCluster);
List<KubernetesClusterVmMapVO> vmList =
kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
// Only validate hypervisor compatibility when upgrading existing VMs.
// Horizontal scaling (adding new nodes) on KVM must not be blocked here.
if (tobeScaledVMCount > 0) {
validateKubernetesClusterScaleOfferingParameters();
}
for (long i = 0; i < tobeScaledVMCount; i++) {
KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
boolean result = false;
try {
result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(),
new HashMap<String, String>());
} catch (RuntimeException | ResourceUnavailableException | ManagementServerException
| VirtualMachineMigrationException e) {
logTransitStateAndThrow(Level.ERROR,
String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s",
kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()),
kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
}
if (!result) {
logTransitStateAndThrow(Level.WARN,
String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s",
kubernetesCluster.getName(), userVM.getDisplayName()),
kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN,
String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out",
kubernetesCluster.getName()),
kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering,
updateNodeOffering, updateClusterOffering);
}`
**Status**:
**Confirmed**: Scale-up and scale-down are working correctly on KVM hypervisors.
**Pending**: Testing on other hypervisors is still required.
**Testing Instructions**
If you would like to test this fix, I have attached a patched cloudstack-4.22.0.0.jar file. You can apply it to your CloudStack Management Server using the following steps:
1. Stop the CloudStack management service.
2. Backup your existing JAR file: `cp /usr/share/cloudstack-management/lib/cloudstack-4.22.0.0.jar /usr/share/cloudstack-management/lib/cloudstack-4.22.0.0.jar.bak`
3. Replace the existing JAR file with the provided patched version.
4. Start the CloudStack management service.
**Link for Patched jar file**
[https://github.com/chandrakantrai/cloudstack4.22.0-Patch/blob/main/cloudstack-4.22.0.0.jar](url)
@weizhouapache, thank you for pointing toward the relevant code section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "Pearl1594 (via GitHub)" <gi...@apache.org>.
Pearl1594 commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-4025981933
> > [@chandrakantrai](https://github.com/chandrakantrai) Currently the issue is not reproducible on 4.22 release
> > The API call with the size parameter is working fine
> > cmk scale kubernetescluster id=<> size=2
> > dynamic change of service offering of a running cks cluster or vm is not supported for kvm hypervisor
> > https://docs.cloudstack.apache.org/en/4.22.0.0/adminguide/virtual_machines.html#cpu-and-memory-scaling-for-running-instances
> > This is the reason you are hitting the exception Scaling Kubernetes cluster : test failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
> > cc [@weizhouapache](https://github.com/weizhouapache) [@sureshanaparti](https://github.com/sureshanaparti)
>
> [@kiranchavala](https://github.com/kiranchavala) I totally agree with you that scaling vm with fixed offering is not supported in KVM.
>
> however, when scale a K8S, scaling often means the provision of new VMs. I am not sure if the inconsistent service offerings (e.g. user specify the service offering when scale Kubernetes cluster) of existing K8S nodes should block the scaling. IMHO, it should apply only on the new VMs.
@weizhouapache I believe this behaviour has been historic. If one attempts to change the offering while changing the cluster size, it changes the offering on the pre-existing nodes as well (not just the new nodes that get added) - Tested this on 4.20.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3833577157
> [@chandrakantrai](https://github.com/chandrakantrai) Currently the issue is not reproducible on 4.22 release
>
> The API call with the size parameter is working fine
>
> cmk scale kubernetescluster id=<> size=2
>
> dynamic change of service offering of a running cks cluster or vm is not supported for kvm hypervisor
>
> https://docs.cloudstack.apache.org/en/4.22.0.0/adminguide/virtual_machines.html#cpu-and-memory-scaling-for-running-instances
>
> This is the reason you are hitting the exception Scaling Kubernetes cluster : test failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
>
> cc [@weizhouapache](https://github.com/weizhouapache) [@sureshanaparti](https://github.com/sureshanaparti)
@kiranchavala
I totally agree with you that scaling vm with fixed offering is not supported in KVM.
however, when scale a K8S, scaling often means the provision of new VMs.
I am not sure if the inconsistent service offerings (e.g. user specify the service offering when scale Kubernetes cluster) of existing K8S nodes should block the scaling. IMHO, it should apply only on the new VMs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-4476606529
Thanks @shwstppr
I think we can add a note in the documentation of the kubernetes page
Cc @sudo87 @Pearl1594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-4523011903
Hello, all
When implementing #12975, I forgot to remove [this constraint](https://github.com/apache/cloudstack/issues/11994#issuecomment-3495510369) in the Kubernetes cluster scaling workflow that prevented changing cluster compute offerings while they were running. I've just opened the PR #13226 to address that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3833493244
@chandrakantrai Currently the issue is not reproducible on 4.22 release
The API call with the size parameter is working fine
cmk scale kubernetescluster id=<> size=2
dynamic change of service offering of a running cks cluster or vm is not supported for kvm hypervisor
https://docs.cloudstack.apache.org/en/4.22.0.0/adminguide/virtual_machines.html#cpu-and-memory-scaling-for-running-instances
This is the reason you are hitting the exception
Scaling Kubernetes cluster : test failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
cc @weizhouapache @sureshanaparti
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "amitmvyas (via GitHub)" <gi...@apache.org>.
amitmvyas commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3495267094
```
kubectl --insecure-skip-tls-verify=true --kubeconfig kube.conf get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7bfdc5b57c-kj99k 1/1 Running 1 (19h ago) 23h
kube-system calico-node-d8dsj 1/1 Running 1 (19h ago) 23h
kube-system calico-node-hqlxb 1/1 Running 1 (19h ago) 23h
kube-system calico-node-lpnrl 1/1 Running 1 (19h ago) 23h
kube-system cloud-controller-manager-857cd68cf5-8xbfr 1/1 Running 2 (19h ago) 23h
kube-system cluster-autoscaler-744b9d65b-zzw2p 1/1 Running 2 (19h ago) 19h
kube-system coredns-674b8bbfcf-9wgdv 1/1 Running 1 (19h ago) 23h
kube-system coredns-674b8bbfcf-s5v82 1/1 Running 1 (19h ago) 23h
kube-system etcd-cl8s-control-19a5261efb6 1/1 Running 1 (19h ago) 23h
kube-system kube-apiserver-cl8s-control-19a5261efb6 1/1 Running 1 (19h ago) 23h
kube-system kube-controller-manager-cl8s-control-19a5261efb6 1/1 Running 1 (19h ago) 23h
kube-system kube-proxy-g98w9 1/1 Running 1 (19h ago) 23h
kube-system kube-proxy-kg5t4 1/1 Running 1 (19h ago) 23h
kube-system kube-proxy-prtkk 1/1 Running 1 (19h ago) 23h
kube-system kube-scheduler-cl8s-control-19a5261efb6 1/1 Running 1 (19h ago) 23h
kubernetes-dashboard dashboard-metrics-scraper-5bd45c9dd6-fm7wn 1/1 Running 1 (19h ago) 23h
kubernetes-dashboard kubernetes-dashboard-687457b9bd-pcdt7 1/1 Running 1 (19h ago) 23h
scale-test big-request-697b5d9654-hscsm 1/1 Running 1 (19h ago) 20h
scale-test big-request-697b5d9654-p224c 0/1 Pending 0 20h
scale-test big-request-697b5d9654-tw7fq 1/1 Running 1 (19h ago) 20h
```
Tried with CMK as well
```
root@innmi1vcsmgt-d001:~# cmk scale kubernetescluster id=da626491-7457-4106-a165-de0d41d2417a size=4
{
"account": "admin",
"accountid": "9b2313ee-b4ae-11f0-914b-bc2411c796cd",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2025-11-06T11:38:18+0530",
"created": "2025-11-06T11:38:18+0530",
"domainid": "abbdc016-b4ad-11f0-914b-bc2411c796cd",
"domainpath": "ROOT",
"jobid": "62e801e2-cc16-4014-9b50-d78b67e875e7",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "9b23987d-b4ae-11f0-914b-bc2411c796cd"
}
🙈 Error: async API failed for job 62e801e2-cc16-4014-9b50-d78b67e875e7
```
This is the logs for cluster-autoscaler.
```
I1106 02:53:19.477668 1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=scaleKubernetesCluster&id=da626491-7457-4106-a165-de0d41d2417a&response=json&size=3&signature=***
I1106 02:53:19.477890 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"ad1efd21-9a52-4226-9c1d-15396820bba5", APIVersion:"v1", ResourceVersion:"205055", FieldPath:""}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group da626491-7457-4106-a165-de0d41d2417a size to 3 instead of 2 (max: 4)
I1106 02:53:19.556058 1 client.go:175] NewAPIRequest response status code:200
I1106 02:53:19.863209 1 reflector.go:255] Listing and watching *v1beta1.PodDisruptionBudget from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309
W1106 02:53:19.865022 1 reflector.go:324] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1106 02:53:19.865056 1 reflector.go:138] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
I1106 02:53:29.557052 1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=queryAsyncJobResult&jobid=acee7bf9-c308-4929-87e5-42f74652d5eb&response=json&signature=***
I1106 02:53:29.597851 1 client.go:175] NewAPIRequest response status code:200
E1106 02:53:29.598077 1 client.go:118] API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresulttype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
W1106 02:53:29.598146 1 clusterstate.go:277] Disabling scale-up for node group da626491-7457-4106-a165-de0d41d2417a until 2025-11-06 03:23:19.477324222 +0000 UTC m=+64348.574537733; errorClass=Other; errorCode=cloudProviderError
I1106 02:53:29.598178 1 metrics.go:380] Function scaleUp took 10.120850343s to complete
I1106 02:53:29.598196 1 eventing_scale_up_processor.go:47] Skipping event processing for unschedulable pods since there is a ScaleUp attempt this loop
E1106 02:53:29.598212 1 static_autoscaler.go:459] Failed to scale up: failed to increase node group size: Unable to scale cluster : API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresulttype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
I1106 02:53:29.598232 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"ad1efd21-9a52-4226-9c1d-15396820bba5", APIVersion:"v1", ResourceVersion:"205055", FieldPath:""}): type: 'Warning' reason: 'FailedToScaleUpGroup' Scale-up failed for group da626491-7457-4106-a165-de0d41d2417a: Unable to scale cluster : API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresultt
ype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
I1106 02:53:29.613991 1 metrics.go:380] Function main took 10.194459637s to complete
I1106 02:53:30.318148 1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StatefulSet total 8 items received
I1106 02:53:36.016083 1 reflector.go:255] Listing and watching *v1beta1.CSIStorageCapacity from k8s.io/client-go/informers/factory.go:134
W1106 02:53:36.018008 1 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource
E1106 02:53:36.018094 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource
I1106 02:53:36.329876 1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 11 items received
I1106 02:53:39.614541 1 static_autoscaler.go:235] Starting main loop
I1106 02:53:39.614764 1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=listKubernetesClusters&id=da626491-7457-4106-a165-de0d41d2417a&response=json&signature=***
I1106 02:53:39.679225 1 client.go:175] NewAPIRequest response status code:200
```
I have attached entire logs as well.
[cluster-autoscaler-744b9d65b-zzw2p.log](https://github.com/user-attachments/files/23382972/cluster-autoscaler-744b9d65b-zzw2p.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "fdegner (via GitHub)" <gi...@apache.org>.
fdegner commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-4933090996
Good morning,
autoscaling of Kubernetes clusters on KVM fails when more than a single service offering is used in a cluster.
The cluster-autoscaler calls `ScaleKubernetesClusterCmd` with only a new `clustersize` (no offering). `createNodeTypeToServiceOfferingMap()` injects a single global `DEFAULT`, and `scaleCluster()` then compares every node type's real VM offering against that global default instead of against that node type's own stored offering (`control_node_service_offering_id` / `worker_node_service_offering_id`).
This leads to a code path were `scaleKubernetesClusterOffering() -> validateKubernetesClusterScaleOfferingParameters()` are called, subsequently triggering the hypervisor type check which then fails and leads to the observed error.
[plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java](https://github.com/apache/cloudstack/blob/348ce953a99246a756b527994f7745a7be038234/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java#L547)
```java
public boolean scaleCluster() throws CloudRuntimeException {
...
for (KubernetesClusterNodeType nodeType : Arrays.asList(CONTROL, ETCD, WORKER)) {
...
// This is were things go wrong
boolean isNodeOfferingScalingNeeded = isServiceOfferingScalingNeededForNodeType(existingServiceOffering, scalingServiceOffering);
...
if (isNodeOfferingScalingNeeded && clusterSizeScalingNeeded) {
if (newVMRequired > 0) {
scaleKubernetesClusterOffering(nodeType, scalingServiceOffering, updateNodeOffering, updateClusterOffering);
scaleKubernetesClusterSize(nodeType);
} else {
scaleKubernetesClusterSize(nodeType);
scaleKubernetesClusterOffering(nodeType, scalingServiceOffering, updateNodeOffering, updateClusterOffering);
}
} else if (isNodeOfferingScalingNeeded) {
scaleKubernetesClusterOffering(nodeType, scalingServiceOffering, updateNodeOffering, updateClusterOffering);
} else if (clusterSizeScalingNeeded) {
scaleKubernetesClusterSize(nodeType); // We should end up here instead!
}
}
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3495510369
the whole block was introduced in the original PR #3680
https://github.com/apache/cloudstack/blob/dbda673e1fa813856deb0f0b6328dad0222b702c/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java#L316-L324
cc @shwstppr
do you remember why KVM is not supported ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3833873835
Thanks @weizhouapache
There is an existing improvement request that I logged to support Custom compute offerings for CKS deployments
https://github.com/apache/cloudstack/issues/11545
@abh1sar @sureshanaparti @rajujith Can we consider for the upcoming releases ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "shwstppr (via GitHub)" <gi...@apache.org>.
shwstppr commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-4476564278
Just mentioning here as I read issue could be with scaling with fixed offering on KVM, https://github.com/apache/cloudstack/pull/12975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3495655416
> [@weizhouapache](https://github.com/weizhouapache) that check was for scaling the cluster with a new offering. It shouldn't affect scaling for node count unless something changed recently. Scaling running VM may not be working on KVM at the time of [#3680](https://github.com/apache/cloudstack/pull/3680)
thanks @shwstppr , makes sense.
so the issue could be related to #9102. cc @nvazquez @Pearl1594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "Jayd603 (via GitHub)" <gi...@apache.org>.
Jayd603 commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3493547180
I can't imagine why scaling k8s (by adding nodes right?) would not work on KVM. That sounds ridiculous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Unable to Autoscale K8s in CS 4.21.0.0 [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3494977701
@amitmvyas
Did you enable autoscaling on a kubernetes cluster which is in running state
<img width="816" height="617" alt="Image" src="https://github.com/user-attachments/assets/237bb816-fb15-458a-bb0a-6d4d568c6bf5" />
Is the autoscaler pod running in the kubernetes cluster , can you provide the logs from the autoscaler pod
```
k get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7bfdc5b57c-nnzm4 1/1 Running 0 2m36s
kube-system calico-node-7qxkb 1/1 Running 0 2m32s
kube-system calico-node-lh5tk 1/1 Running 0 2m36s
kube-system cloud-controller-manager-857cd68cf5-p5tvc 1/1 Running 0 110s
kube-system cluster-autoscaler-6d469c6c6-x6f9m 1/1 Running 0 40s
```
Example
k logs -f cluster-autoscaler-6d469c6c6-x6f9m -n kube-system
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org