You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "ahmadamirahmadi1401 (via GitHub)" <gi...@apache.org> on 2025/06/11 14:25:41 UTC
[D] Upgrading K8s Cluster Failed [cloudstack]
GitHub user ahmadamirahmadi1401 created a discussion: Upgrading K8s Cluster Failed
CloudStack 4.19.1.3
My upgrade from K8S Version 1.31.8 into 1.32.4 was unsuccessful.
2025-06-03 07:42:57,510 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
GitHub link: https://github.com/apache/cloudstack/discussions/11006
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Upgrading K8s Cluster Failed
> Why does the middle log say the upgrade was successful, but the next log says the upgrade failed?
>
> What do you think is the problem? Do you have a solution to suggest?
>
the process has some steps, including the upgrade of kuberbetes clusters.
it looks like kubernetes cluster has been upgraded successfully, there are an issue in the post-upgrade step.
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13434097
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "ahmadamirahmadi1401 (via GitHub)" <gi...@apache.org>.
GitHub user ahmadamirahmadi1401 added a comment to the discussion: Upgrading K8s Cluster Failed
@weizhouapache
These have been repeated over and over again.
2025-06-03 07:42:57,510 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
unpacking k8s.gcr.io/sig-storage/csi-attacher:v3.0.2 (sha256:6f80b12657a7e0a5c683b24e806c4bbbe33a43e39b041fe9b7514d665d478ea4)...done
unpacking registry.k8s.io/sig-storage/csi-attacher:v4.6.1 (sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94)...done
unpacking k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1 (sha256:e07f914c32f0505e4c470a62a40ee43f84cbf8dc46ff861f31b14457ccbad108)...done
unpacking registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1 (sha256:f25af73ee708ff9c82595ae99493cdef9295bd96953366cddf36305f82555dac)...done
unpacking k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4 (sha256:bec571992d40203edcd056ac0b0d97003887ee5e4be144c41932d18639673b03)...done
unpacking registry.k8s.io/sig-storage/csi-provisioner:v5.0.1 (sha256:405a14e1aa702f7ea133cea459e8395fe40a6125c088c55569e696d48e1bd385)...done
unpacking registry.k8s.io/sig-storage/csi-resizer:v1.11.1 (sha256:a541e6cc2d8b011bb21b1d4ffec6b090e85270cce6276ee302d86153eec0af43)...done
unpacking docker.io/kubernetesui/dashboard:v2.7.0 (sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93)...done
unpacking registry.k8s.io/etcd:3.5.16-0 (sha256:c6a9d11cc5c04b114ccdef39a9265eeef818e3d02f5359be035ae784097fdec5)...done
unpacking registry.k8s.io/kube-apiserver:v1.32.4 (sha256:631c6cc78b2862be4fed7df3384a643ef7297eebadae22e8ef9cbe2e19b6386f)...done
unpacking registry.k8s.io/kube-controller-manager:v1.32.4 (sha256:25e29187ea66f0ff9b9a00114849c3a30b649005c900a8b2a69e3f3fa56448fb)...done
unpacking registry.k8s.io/kube-proxy:v1.32.4 (sha256:152638222ecf265eb8e5352e3c50e8fc520994e8ffcff1ee1490c975f7fc2b36)...done
unpacking registry.k8s.io/kube-scheduler:v1.32.4 (sha256:09c55f8dac59a4b8e5e354140f5a4bdd6fa9bd95c42d6bcba6782ed37c31b5a2)...done
unpacking registry.k8s.io/sig-storage/livenessprobe:v2.12.0 (sha256:5baeb4a6d7d517434292758928bb33efc6397368cbb48c8a4cf29496abf4e987)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.8 (sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c)...done
unpacking registry.k8s.io/pause:3.10 (sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a)...done
unpacking registry.mainco.com/weaveworks/weave-kube:latest (sha256:35827a9c549c095f0e9d1cf8b35d8f27ae2c76e31bc6f7f3c0bc95911d5accea)...done
unpacking registry.mainco.com/weaveworks/weave-npc:latest (sha256:062832fd25b5e9e16650e618f26bba1409a7b3bf2c3903e1b369d788abc63aef)...done
registry.k8s.io/pause:3.10
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
========
2025-06-03 07:42:57,511 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) Failed to upgrade Kubernetes cluster : 82c623a8ac92d1ffee84962c976e6800-01jwtbe6dpqkmw633gwkaagpba, unable to upgrade Kubernetes node on VM : k8s-82c623a8ac92d1ffee84962c976e6800-01j-control-19734b7e1d6, retries left: 3
=======
2025-06-03 07:43:55,561 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
=====
2025-06-03 07:43:56,433 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
mount: /mnt/k8sdisk: /dev/sr0 already mounted on /mnt/k8sdisk.
cp: cannot create regular file '/opt/bin/kubeadm': Text file busy
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13434092
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "ahmadamirahmadi1401 (via GitHub)" <gi...@apache.org>.
GitHub user ahmadamirahmadi1401 added a comment to the discussion: Upgrading K8s Cluster Failed
its another failed in upgrafing from 1.30.5 into 1.30.13
2025-06-07 14:03:11,554 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-88:ctx-2e915bda job-33084 ctx-196a511a) (logid:16a30b3c) Failed to upgrade Kubernetes cluster : 22af645d1859cb5ca6da0c484f1f37ea-01jx57z4hpk29trtvk84bk664k, unable to upgrade Kubernetes node on VM : k8s-22af645d1859cb5ca6da0c484f1f37ea-01j-control-1974a805516
com.cloud.utils.exception.CloudRuntimeException: Failed to upgrade Kubernetes cluster : 22af645d1859cb5ca6da0c484f1f37ea-01jx57z4hpk29trtvk84bk664k, unable to upgrade Kubernetes node on VM : k8s-22af645d1859cb5ca6da0c484f1f37ea-01j-control-1974a805516
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.logTransitStateDetachIsoAndThrow(KubernetesClusterActionWorker.java:266)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeKubernetesClusterNodes(KubernetesClusterUpgradeWorker.java:127)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeCluster(KubernetesClusterUpgradeWorker.java:177)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.upgradeKubernetesCluster(KubernetesClusterManagerImpl.java:1622)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy466.upgradeKubernetesCluster(Unknown Source)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd.execute(UpgradeKubernetesClusterCmd.java:114)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13434095
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Upgrading K8s Cluster Failed
are there more logs ?
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13434091
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "rafaelpagliari (via GitHub)" <gi...@apache.org>.
GitHub user rafaelpagliari added a comment to the discussion: Upgrading K8s Cluster Failed
Any updates on this?
I'm having the same issue. I have two ACS with the same version (4.18). One has the upgrade successfully, the other has the same issue.
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13936313
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "ahmadamirahmadi1401 (via GitHub)" <gi...@apache.org>.
GitHub user ahmadamirahmadi1401 added a comment to the discussion: Upgrading K8s Cluster Failed
Why does the middle log say the upgrade was successful, but the next log says the upgrade failed?
What do you think is the problem? Do you have a solution to suggest?
2025-06-08 07:14:30,886 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-6:ctx-559bc6d5 job-33225 ctx-3b4cbb1d) (logid:33c83a6c) SSH execution of command sudo ./upgrade-kubernetes.sh4071027720196768380.sh 1.30.13 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
unpacking k8s.gcr.io/sig-storage/csi-attacher:v3.0.2 (sha256:6f80b12657a7e0a5c683b24e806c4bbbe33a43e39b041fe9b7514d665d478ea4)...done
unpacking registry.k8s.io/sig-storage/csi-attacher:v4.6.1 (sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94)...done
unpacking k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1 (sha256:e07f914c32f0505e4c470a62a40ee43f84cbf8dc46ff861f31b14457ccbad108)...done
unpacking registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1 (sha256:f25af73ee708ff9c82595ae99493cdef9295bd96953366cddf36305f82555dac)...done
unpacking k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4 (sha256:bec571992d40203edcd056ac0b0d97003887ee5e4be144c41932d18639673b03)...done
unpacking registry.k8s.io/sig-storage/csi-provisioner:v5.0.1 (sha256:405a14e1aa702f7ea133cea459e8395fe40a6125c088c55569e696d48e1bd385)...done
unpacking registry.k8s.io/sig-storage/csi-resizer:v1.11.1 (sha256:a541e6cc2d8b011bb21b1d4ffec6b090e85270cce6276ee302d86153eec0af43)...done
unpacking docker.io/kubernetesui/dashboard:v2.7.0 (sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93)...done
unpacking registry.k8s.io/etcd:3.5.15-0 (sha256:a6dc63e6e8cfa0307d7851762fa6b629afb18f28d8aa3fab5a6e91b4af60026a)...done
unpacking registry.k8s.io/kube-apiserver:v1.30.13 (sha256:bd68d81c20ad5781adec9f6eae24c83d6f66c3afc0d4baa32d3f2d865c82d436)...done
unpacking registry.k8s.io/kube-controller-manager:v1.30.13 (sha256:b85a5d785cc00b03613f67a7afca1e91b2a0ccf4ced8e69d38bda5f686980f3f)...done
unpacking registry.k8s.io/kube-proxy:v1.30.13 (sha256:f68590d1921db1f9a9b56898a66cf3c67fcdcb5be132f7640af8ac1297371e4a)...done
unpacking registry.k8s.io/kube-scheduler:v1.30.13 (sha256:bfa5f10dc3e14316785f23f8e1f4e4dbc556b033270d91185fca0577353cad33)...done
unpacking registry.k8s.io/sig-storage/livenessprobe:v2.12.0 (sha256:5baeb4a6d7d517434292758928bb33efc6397368cbb48c8a4cf29496abf4e987)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.8 (sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c)...done
unpacking registry.k8s.io/pause:3.9 (sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097)...done
unpacking registry.mainxcloud.com/weaveworks/weave-kube:latest (sha256:35827a9c549c095f0e9d1cf8b35d8f27ae2c76e31bc6f7f3c0bc95911d5accea)...done
unpacking registry.mainxcloud.com/weaveworks/weave-npc:latest (sha256:062832fd25b5e9e16650e618f26bba1409a7b3bf2c3903e1b369d788abc63aef)...done
registry.k8s.io/pause:3.9
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.30.13"
[upgrade/versions] Cluster version: v1.30.13
[upgrade/versions] kubeadm version: v1.30.13
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.30.13" (timeout: 5m0s)...
[upgrade/etcd] Upgrading to TLS for etcd
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Current and new manifests of etcd are equal, skipping upgrade
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4014411918"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Current and new manifests of kube-apiserver are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Current and new manifests of kube-controller-manager are equal, skipping upgrade
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Current and new manifests of kube-scheduler are equal, skipping upgrade
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1007267359/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.30.13". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
serviceaccount/weave-net unchanged
clusterrole.rbac.authorization.k8s.io/weave-net unchanged
clusterrolebinding.rbac.authorization.k8s.io/weave-net unchanged
role.rbac.authorization.k8s.io/weave-net unchanged
rolebinding.rbac.authorization.k8s.io/weave-net unchanged
daemonset.apps/weave-net configured
serviceaccount/cloudstack-csi-controller unchanged
clusterrole.rbac.authorization.k8s.io/cloudstack-csi-controller-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/cloudstack-csi-controller-binding unchanged
csidriver.storage.k8s.io/csi.cloudstack.apache.org unchanged
deployment.apps/cloudstack-csi-controller unchanged
daemonset.apps/cloudstack-csi-node unchanged
mount: /mnt/k8sdisk: /dev/sr0 already mounted on /mnt/k8sdisk.
2025-06-08 07:13:42,139 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-6:ctx-559bc6d5 job-33225 ctx-3b4cbb1d) (logid:33c83a6c) Failed to upgrade Kubernetes cluster : c, unable to upgrade Kubernetes node on VM : c-control-1974e576a15, retries left: 2
@weizhouapache @DaanHoogland
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13434096
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "Pearl1594 (via GitHub)" <gi...@apache.org>.
GitHub user Pearl1594 added a comment to the discussion: Upgrading K8s Cluster Failed
what is the output of `kubectl get nodes` for the cluster, just to see if the nodes have actually upgraded, but if an issue happened post upgrade.
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13936413
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "rafaelpagliari (via GitHub)" <gi...@apache.org>.
GitHub user rafaelpagliari added a comment to the discussion: Upgrading K8s Cluster Failed
I'm only looking at the logs below; at first glance, I don't see an error that would explain exactly why this is happening.
For clarification, the tests on both ACS are with the clusters at zero. That is, I simply created the cluster and sent it to upgrade both.
Immediately after the log below, the cluster enters alert status in ACS, and I receive the error. These logs are repeated many times afterward, so I believe there is no direct correlation.
Jul 30 16:36:21 systemvm kubelet[5482]: I0730 16:36:21.218040 5482 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/weave-net-szlwg" podStartSLOduration=5.217763853 podCreationTimestamp="2025-07-30 16:36:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-07-30 16:36:20.231952365 +0000 UTC m=+82.846380214" watchObservedRunningTime="2025-07-30 16:36:21.217763853 +0000 UTC m=+83.832191726"
Jul 30 16:36:57 systemvm containerd[5348]: time="2025-07-30T16:36:57.665646046Z" level=info msg="StopPodSandbox for \"7e6e4a443c74cf7f00e0f79c8aa4b4e6f194abf8ca0aad013a1d8da0bc8134c3\""
Jul 30 16:36:57 systemvm containerd[5348]: time="2025-07-30T16:36:57.666075939Z" level=info msg="TearDown network for sandbox \"7e6e4a443c74cf7f00e0f79c8aa4b4e6f194abf8ca0aad013a1d8da0bc8134c3\" successfully"
Jul 30 16:36:57 systemvm containerd[5348]: time="2025-07-30T16:36:57.666220832Z" level=info msg="StopPodSandbox for \"7e6e4a443c74cf7f00e0f79c8aa4b4e6f194abf8ca0aad013a1d8da0bc8134c3\" returns successfully"
Jul 30 16:36:57 systemvm containerd[5348]: time="2025-07-30T16:36:57.666982916Z" level=info msg="RemovePodSandbox for \"7e6e4a443c74cf7f00e0f79c8aa4b4e6f194abf8ca0aad013a1d8da0bc8134c3\""
Jul 30 16:36:57 systemvm containerd[5348]: time="2025-07-30T16:36:57.667049075Z" level=info msg="Forcibly stopping sandbox \"7e6e4a443c74cf7f00e0f79c8aa4b4e6f194abf8ca0aad013a1d8da0bc8134c3\""
Jul 30 16:36:57 systemvm containerd[5348]: time="2025-07-30T16:36:57.667230536Z" level=info msg="TearDown network for sandbox \"7e6e4a443c74cf7f00e0f79c8aa4b4e6f194abf8ca0aad013a1d8da0bc8134c3\" successfully"
Jul 30 16:36:57 systemvm containerd[5348]: time="2025-07-30T16:36:57.675659342Z" level=info msg="RemovePodSandbox \"7e6e4a443c74cf7f00e0f79c8aa4b4e6f194abf8ca0aad013a1d8da0bc8134c3\" returns successfully"
Jul 30 16:37:28 systemvm systemd[1]: Starting User Manager for UID 1000...
Jul 30 16:37:28 systemvm systemd[7907]: PAM failed: Critical error - immediate abort
Jul 30 16:37:28 systemvm systemd[7907]: user@1000.service: Failed to set up PAM session: Operation not permitted
Jul 30 16:37:28 systemvm systemd[7907]: user@1000.service: Failed at step PAM spawning /lib/systemd/systemd: Operation not permitted
Jul 30 16:37:28 systemvm systemd[1]: user@1000.service: Main process exited, code=exited, status=224/PAM
Jul 30 16:37:28 systemvm systemd[1]: user@1000.service: Failed with result 'exit-code'.
Jul 30 16:37:28 systemvm systemd[1]: Failed to start User Manager for UID 1000.
Jul 30 16:37:28 systemvm systemd[1]: Started Session 33 of user cloud.
Jul 30 16:37:28 systemvm systemd[1]: session-33.scope: Succeeded.
Jul 30 16:37:29 systemvm systemd[1]: Starting User Manager for UID 1000...
Jul 30 16:37:29 systemvm systemd[7928]: PAM failed: Critical error - immediate abort
Jul 30 16:37:29 systemvm systemd[7928]: user@1000.service: Failed to set up PAM session: Operation not permitted
Jul 30 16:37:29 systemvm systemd[7928]: user@1000.service: Failed at step PAM spawning /lib/systemd/systemd: Operation not permitted
Jul 30 16:37:29 systemvm systemd[1]: user@1000.service: Main process exited, code=exited, status=224/PAM
Jul 30 16:37:29 systemvm systemd[1]: user@1000.service: Failed with result 'exit-code'.
Jul 30 16:37:29 systemvm systemd[1]: Failed to start User Manager for UID 1000.
Jul 30 16:37:29 systemvm systemd[1]: Started Session 34 of user cloud.
Jul 30 16:37:29 systemvm systemd[1]: session-34.scope: Succeeded.
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13936688
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "ahmadamirahmadi1401 (via GitHub)" <gi...@apache.org>.
GitHub user ahmadamirahmadi1401 added a comment to the discussion: Upgrading K8s Cluster Failed
Yes, but I did not make any changes to the file and this is the default behavior when upgrading.
This happens for other versions as well. For example, the following error was obtained for upgrading a Kubernetes cluster from version 1.30.5 to version 1.31.8. And in the end, it failed to unmount the ISO and got an alert.
2025-06-03 18:40:17,001 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-34:ctx-a226f7b8 job-32868 ctx-5a7f130c) (logid:f244ee01) SSH execution of command sudo ./upgrade-kubernetes.sh13249113815295454536.sh 1.31.8 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
unpacking k8s.gcr.io/sig-storage/csi-attacher:v3.0.2 (sha256:6f80b12657a7e0a5c683b24e806c4bbbe33a43e39b041fe9b7514d665d478ea4)...done
unpacking registry.k8s.io/sig-storage/csi-attacher:v4.6.1 (sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94)...done
unpacking k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1 (sha256:e07f914c32f0505e4c470a62a40ee43f84cbf8dc46ff861f31b14457ccbad108)...done
unpacking registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1 (sha256:f25af73ee708ff9c82595ae99493cdef9295bd96953366cddf36305f82555dac)...done
unpacking k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4 (sha256:bec571992d40203edcd056ac0b0d97003887ee5e4be144c41932d18639673b03)...done
unpacking registry.k8s.io/sig-storage/csi-provisioner:v5.0.1 (sha256:405a14e1aa702f7ea133cea459e8395fe40a6125c088c55569e696d48e1bd385)...done
unpacking registry.k8s.io/sig-storage/csi-resizer:v1.11.1 (sha256:a541e6cc2d8b011bb21b1d4ffec6b090e85270cce6276ee302d86153eec0af43)...done
unpacking docker.io/kubernetesui/dashboard:v2.7.0 (sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93)...done
unpacking registry.k8s.io/etcd:3.5.15-0 (sha256:a6dc63e6e8cfa0307d7851762fa6b629afb18f28d8aa3fab5a6e91b4af60026a)...done
unpacking registry.k8s.io/kube-apiserver:v1.31.8 (sha256:30090db6a7d53799163ce82dae9e8ddb645fd47db93f2ec9da0cc787fd825625)...done
unpacking registry.k8s.io/kube-controller-manager:v1.31.8 (sha256:29eaddc64792a689df48506e78bbc641d063ac8bb92d2e66ae2ad05977420747)...done
unpacking registry.k8s.io/kube-proxy:v1.31.8 (sha256:dd0c9a37670f209947b1ed880f06a2e93e1d41da78c037f52f94b13858769838)...done
unpacking registry.k8s.io/kube-scheduler:v1.31.8 (sha256:22994a2632e81059720480b9f6bdeb133b08d58492d0b36dfd6e9768b159b22a)...done
unpacking registry.k8s.io/sig-storage/livenessprobe:v2.12.0 (sha256:5baeb4a6d7d517434292758928bb33efc6397368cbb48c8a4cf29496abf4e987)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.8 (sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c)...done
unpacking registry.k8s.io/pause:3.10 (sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a)...done
unpacking registry.mainx.com/weaveworks/weave-kube:latest (sha256:35827a9c549c095f0e9d1cf8b35d8f27ae2c76e31bc6f7f3c0bc95911d5accea)...done
unpacking registry.mainx.com/weaveworks/weave-npc:latest (sha256:062832fd25b5e9e16650e618f26bba1409a7b3bf2c3903e1b369d788abc63aef)...done
registry.k8s.io/pause:3.9
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
====================
2025-06-03 18:43:04,106 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-34:ctx-a226f7b8 job-32868 ctx-5a7f130c) (logid:f244ee01) Failed to upgrade Kubernetes cluster : e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable to upgrade Kubernetes node on VM : e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089
com.cloud.utils.exception.CloudRuntimeException: Failed to upgrade Kubernetes cluster : e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable to upgrade Kubernetes node on VM : e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.logTransitStateDetachIsoAndThrow(KubernetesClusterActionWorker.java:266)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeKubernetesClusterNodes(KubernetesClusterUpgradeWorker.java:127)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeCluster(KubernetesClusterUpgradeWorker.java:177)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.upgradeKubernetesCluster(KubernetesClusterManagerImpl.java:1622)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy466.upgradeKubernetesCluster(Unknown Source)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd.execute(UpgradeKubernetesClusterCmd.java:114)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
=============================
2025-06-03 18:43:04,685 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-34:ctx-a226f7b8 job-32868) (logid:f244ee01) Complete async job-32868, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to upgrade Kubernetes cluster : e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable to upgrade Kubernetes node on VM : e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089"}
@weizhouapache
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13434094
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "rafaelpagliari (via GitHub)" <gi...@apache.org>.
GitHub user rafaelpagliari added a comment to the discussion: Upgrading K8s Cluster Failed
cloud@k8s-acs1-control-1985b69a62a:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-acs1-control-1985b69a62a Ready control-plane 52m v1.28.4
k8s-acs1-node-1985b6a1fae Ready <none> 52m v1.28.4
k8s-acs1-node-1985b6a92a2 Ready,SchedulingDisabled <none> 52m v1.27.3
k8s-acs1-node-1985b6b0fc4 Ready <none> 52m v1.27.3
It always updates control + node1, it no longer updates node2 and gives the unable to update.
This is the scenario on my ACS1... It updated successfully on ACS2.
They have the same k8s version and the same ACS version.
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13936448
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "Pearl1594 (via GitHub)" <gi...@apache.org>.
GitHub user Pearl1594 added a comment to the discussion: Upgrading K8s Cluster Failed
Do you see any specific failure log pertaining to the upgrade process in `/var/log/daemon.log` - which could be helpful to pinpoint where the issue lies..
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13936513
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Upgrading K8s Cluster Failed
it looks like the issue is caused by
```
cp: cannot create regular file '/opt/bin/kubeadm': Text file busy
```
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13434093
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Upgrading K8s Cluster Failed
@ahmadamirahmadi1401 any updates on this?
GitHub link: https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13641227
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Upgrading K8s Cluster Failed [cloudstack]
Posted by "ahmadamirahmadi1401 (via GitHub)" <gi...@apache.org>.
GitHub user ahmadamirahmadi1401 closed a discussion: Upgrading K8s Cluster Failed
CloudStack 4.19.1.3
My upgrade from K8S Version 1.31.8 into 1.32.4 was unsuccessful.
2025-06-03 07:42:57,510 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-116:ctx-c60f694f job-32763 ctx-b2ed3f15) (logid:ae643eb7) SSH execution of command sudo ./upgrade-kubernetes.sh14647125368844262123.sh 1.32.4 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2 (sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1 (sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0 (sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3 (sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
GitHub link: https://github.com/apache/cloudstack/discussions/11006
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org