You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@cloudstack.apache.org by "baltazorbest (via GitHub)" <gi...@apache.org> on 2025/10/02 14:08:16 UTC

[I] CKS Firewall and scaling cluster problem if default firewall rules delete [cloudstack]

baltazorbest opened a new issue, #11779:
URL: https://github.com/apache/cloudstack/issues/11779

   ### problem
   
   After creating k8s cluster and remove default firewall rules, I cannot scaling cluster, with network error:
   
   > 2025-10-02 13:02:14,919 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-36:[ctx-11ddf09f, job-9391, ctx-04c8b6be, ctx-35bd1fab, ctx-42198853]) (logid:2fbf4611) Skipping delivering event Event {"description":"{\"event\":\"VM.START\",\"status\":\"Completed\"}","eventId":null,"eventType":"VM.START","eventUuid
   ":null,"resourceType":"VirtualMachine","resourceUUID":null} to any webhook as account ID is missing
   2025-10-02 13:02:14,919 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-36:[ctx-11ddf09f, job-9391, ctx-04c8b6be, ctx-35bd1fab, ctx-42198853]) (logid:2fbf4611) Failed to publish event [category: ActionEvent, type: VM.START] on bus webhookEventBus
   2025-10-02 13:02:14,936 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-36:[ctx-11ddf09f, job-9391, ctx-04c8b6be]) (logid:2fbf4611) Scaling failed for Kubernetes cluster : my-k8s, unable to update network rules com.cloud.exception.ManagementServerException: Firewall rule for node SSH access can't
    be provisioned
           at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterIsolatedNetworkRules(KubernetesClusterScaleWorker.java:128)
           at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterNetworkRules(KubernetesClusterScaleWorker.java:176)
           at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleUpKubernetesClusterSize(KubernetesClusterScaleWorker.java:388)
           at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterSize(KubernetesClusterScaleWorker.java:424)
           at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClusterScaleWorker.java:477)
           at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterManagerImpl.java:1767)
           at jdk.internal.reflect.GeneratedMethodAccessor1219.invoke(Unknown Source)
           at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.base/java.lang.reflect.Method.invoke(Method.java:569)
           at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
           at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
           at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
           at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
           at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
           at jdk.proxy3/jdk.proxy3.$Proxy517.scaleKubernetesCluster(Unknown Source)
           at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKubernetesClusterCmd.java:160)
           at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
           at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
           at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
           at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
           at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
           at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
           at java.base/java.lang.Thread.run(Thread.java:840)
   
   ### versions
   
   OS is ubuntu 22.04
   Cloudstack version is 4.20.1
   K8s version is v1.33.1-calico-x86_64
   Primary storage is Ceph RBD 19.2.3
   Libvirt version is 8.0.0-1ubuntu7.12
   
   
   
   ### The steps to reproduce the bug
   
   1. Create a network with any subnet (e.g., 10.10.10.1/24).
   2. Create a k8s cluster in HA mode with one worker node, using the previously created external network.
   3. Remove the default firewall rules:
   
   - 0.0.0.0/0 TCP 6443 6443
   - 0.0.0.0/0 TCP 2222 2225
   
   4. Add new firewall rules:
   
   - 10.10.10.1/24 TCP 1 65534
   - 1.1.1.1/32 TCP 1 65534
   
   5. Try to scale the cluster to two worker nodes.
   
   Result:
   An error occurs, although the new instance is created.
   
   Workaround
   When using the following firewall rules instead:
   
   - 10.10.10.1/24 TCP 6443 6443
   - 10.10.10.1/24 TCP 2222 2225
   
   → Scaling the cluster works correctly.
   
   Additional Issues Observed
   1. Opening SSH (2222–2225) and k8s management (6443) to 0.0.0.0/0 is a security risk.
   2. When k8s enters the alert state, it is impossible to repair the cluster. The only options available are stop or delete.
   - After stopping and starting the cluster, it change state to running. The new worker instance is created, but it does not join the Kubernetes cluster (it isn’t present in the cluster).
   - However, scaling the cluster is still not possible, and deleting an individual instance also fails.
   - The only option left is to remove the entire cluster and create it again.
   
   
   
   ### What to do about it?
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] CKS Firewall and scaling cluster problem if default firewall rules delete [cloudstack]

Posted by "baltazorbest (via GitHub)" <gi...@apache.org>.

baltazorbest commented on issue #11779:
URL: https://github.com/apache/cloudstack/issues/11779#issuecomment-3364453571

   @weizhouapache Hey, thanks for your reply. What do you think about the security risk with the default firewall rules? I mean opening 0.0.0.0/0 for ports 6443 and 2222–22xx.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] CKS Firewall and scaling cluster problem if default firewall rules delete [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.

DaanHoogland closed issue #11779: CKS Firewall and scaling cluster problem if default firewall rules delete
URL: https://github.com/apache/cloudstack/issues/11779


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] CKS Firewall and scaling cluster problem if default firewall rules delete [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.

weizhouapache commented on issue #11779:
URL: https://github.com/apache/cloudstack/issues/11779#issuecomment-3361511780

   https://github.com/apache/cloudstack/issues/11758#issuecomment-3356000079


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org