You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "hodie-aurora (via GitHub)" <gi...@apache.org> on 2025/08/12 03:28:20 UTC
[I] Bug: CKS HA cluster initialization fails, only one port forwarding rule is accessible [cloudstack]
hodie-aurora opened a new issue, #11431:
URL: https://github.com/apache/cloudstack/issues/11431
### problem
Issue Type: Bug Report
CloudStack Version:
4.20.x (Based on Virtual Router version 4.20.0)
Hypervisor:
KVM
Kubernetes Template:
setup-v1.33.1-calico-x86_64.iso
Summary:
When creating a high-availability (HA) Kubernetes cluster using the CloudStack Kubernetes Service (CKS) in a VPC, the cluster initialization process gets stuck and eventually times out. The root cause appears to be that only one of the automatically created port forwarding rules for the control plane nodes is functional, preventing the management server and nodes from communicating with each other.
Steps to Reproduce:
Create a new VPC with a single network tier (e.g., 10.1.0.0/24).
Navigate to Compute -> Kubernetes and start creating a new HA cluster.
Select the setup-v1.33.1-calico-x86_64.iso template.
Configure it with 3 control plane nodes and 1 worker node.
Select the VPC network created in step 1.
Crucially, leave the "Load Balancer IP" field empty, allowing CloudStack to automatically acquire a public IP and create the necessary rules.
Launch the cluster.
Expected Results:
The cluster VMs are created, all necessary port forwarding rules (for SSH on port 22 and the K8s API on port 6443) are created on the VPC's virtual router, and the cluster successfully initializes, reaching a "Running" state. All nodes should be accessible via their respective forwarded ports.
Actual Results:
The cluster VMs are created successfully.
All port forwarding rules are listed correctly in the CloudStack UI.
The cluster state remains "Starting" for a long time, with management server logs repeatedly showing: Waiting for Kubernetes cluster ... control node VMs to be accessible.
After a timeout, the cluster enters an "Error" state.
Network tests (telnet <public_ip> <forwarded_port>) reveal that only one of the SSH port forwarding rules works. The others fail with a No route to host error. The single working rule is not always for the same node across different creation attempts.
Diagnostics Performed:
Port Forwarding Test:
basic
# telnet 192.168.10.225 2224 --> Connected successfully
# telnet 192.168.10.225 2222 --> telnet: Unable to connect to remote host: No route to host
# telnet 192.168.10.225 2223 --> telnet: Unable to connect to remote host: No route to host
# telnet 192.168.10.225 2225 --> telnet: Unable to connect to remote host: No route to host
Virtual Router iptables Inspection:
I logged into the VPC's virtual router (r-53-VM) and confirmed that all DNAT rules are present and correct. This proves the issue is not with the virtual router's configuration.
apache
# iptables-save | grep DNAT
-A PREROUTING -d 192.168.10.225/32 -p tcp -m tcp --dport 6443 -j DNAT --to-destination 10.1.0.209:6443
-A PREROUTING -d 192.168.10.225/32 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.1.0.209:22
-A PREROUTING -d 192.168.10.225/32 -p tcp -m tcp --dport 2223 -j DNAT --to-destination 10.1.0.80:22
-A PREROUTING -d 192.168.10.225/32 -p tcp -m tcp --dport 2224 -j DNAT --to-destination 10.1.0.254:22
-A PREROUTING -d 192.168.10.225/32 -p tcp -m tcp --dport 2225 -j DNAT --to-destination 10.1.0.72:22
... (and corresponding OUTPUT chain rules)
Hypothesis:
Since the virtual router is correctly forwarding traffic, the No route to host error strongly suggests that the packets are being rejected by the destination K8s node VMs themselves. The most likely cause is a default-on firewall (like firewalld or ufw) within the setup-v1.33.1-calico-x86_64.iso template. This firewall blocks incoming SSH connections from the virtual router, preventing cluster setup.
The fact that one node is sometimes accessible might be due to timing, where one node's firewall is temporarily disabled during its initial setup phase before the rest of the cluster setup fails.
Request:
Could the development team please investigate if the CKS templates have a firewall enabled by default? If so, this seems to break the HA cluster creation process in a VPC and should either be disabled or pre-configured to allow traffic from the VPC's private network range.
### versions
CloudStack Version: ~4.20.1.0 (inferred from Virtual Router software version)
Hypervisor: KVM
Primary Storage: NFS (inferred from compute offering name 8cpu-16g-nfs)
Network: CloudStack VPC with an Isolated Guest Network and Virtual Router.
CKS Template: setup-v1.33.1-calico-x86_64.iso
### The steps to reproduce the bug
Create a new VPC with a single network tier (e.g., 10.1.0.0/24).
Navigate to Compute -> Kubernetes and start creating a new HA cluster.
Select the setup-v1.33.1-calico-x86_64.iso template.
Configure it with 3 control plane nodes and 1 worker node.
Select the VPC network created in step 1.
Leave the "Load Balancer IP" field empty to allow CloudStack to automatically acquire an IP.
Launch the cluster and observe that it fails to initialize, with only one forwarded port being accessible.
### What to do about it?
Proposed Long-Term Fix:
The CKS template setup-v1.33.1-calico-x86_64.iso should be modified. The firewall inside the template should either be disabled by default or, preferably, be pre-configured with rules that allow all necessary traffic for cluster setup. At a minimum, it should allow inbound traffic from the VPC's private network CIDR (e.g., 10.1.0.0/24 in this case) on the required ports (like SSH port 22 and Kubernetes API port 6443) to ensure the cluster initialization can complete successfully without manual intervention.
Immediate Workaround for Users:
For users encountering this bug, a potential workaround is to use the noVNC console from the CloudStack UI to access each Kubernetes node VM. Once logged in, manually disable the firewall. For example:
On systemd-based systems: systemctl stop firewalld && systemctl disable firewalld
On Debian/Ubuntu systems: ufw disable
After disabling the firewalls on all nodes, the cluster setup process should be able to proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bug: CKS HA cluster initialization fails, only one port forwarding rule is accessible [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on issue #11431:
URL: https://github.com/apache/cloudstack/issues/11431#issuecomment-3183326507
@hodie-aurora, I am unable to reproduce the issue on the latest 4.21 RC3
used
https://download.cloudstack.org/cks/setup-v1.33.1-calico-x86_64.iso
vpc offering > Default VPC offering
vpctier offering > DefaultIsolatedNetworkOfferingForVpcNetworks
ACL > Default allow
```
╰─ kubectl --insecure-skip-tls-verify=true get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
test3-control-198a2de2f22 Ready control-plane 55m v1.33.1 10.1.0.47 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-37-amd64 containerd://1.7.27
test3-control-198a2de762e Ready control-plane 55m v1.33.1 10.1.0.137 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-37-amd64 containerd://1.7.27
test3-control-198a2dec957 Ready control-plane 55m v1.33.1 10.1.0.148 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-37-amd64 containerd://1.7.27
test3-node-198a2df1201 Ready <none> 55m v1.33.1 10.1.0.58 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-37-amd64 containerd://1.7.27
```
<img width="1630" height="681" alt="Image" src="https://github.com/user-attachments/assets/4c1994b0-1e0c-413d-8122-b867adc304cc" />
<img width="1615" height="600" alt="Image" src="https://github.com/user-attachments/assets/8035f089-4b31-4678-8e62-813cca88116e" />
<img width="1626" height="401" alt="Image" src="https://github.com/user-attachments/assets/47a91294-1eb5-4c1e-bf1e-f4dc8423a133" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bug: CKS HA cluster initialization fails, only one port forwarding rule is accessible [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland closed issue #11431: Bug: CKS HA cluster initialization fails, only one port forwarding rule is accessible
URL: https://github.com/apache/cloudstack/issues/11431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Bug: CKS HA cluster initialization fails, only one port forwarding rule is accessible [cloudstack]
Posted by "hodie-aurora (via GitHub)" <gi...@apache.org>.
hodie-aurora commented on issue #11431:
URL: https://github.com/apache/cloudstack/issues/11431#issuecomment-3183346581
@kiranchavala Thank you very much for your help and for taking the time to test this on CloudStack 4.21 RC3. After further investigation, I found that the issue was caused by a problem with my physical switch, not CloudStack itself. The issue has now been resolved. I really appreciate your feedback and assistance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org