You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "mwaag (via GitHub)" <gi...@apache.org> on 2026/06/23 07:45:58 UTC
[D] Kubernetes-dashboard unreachable (sometimes) [cloudstack]
GitHub user mwaag created a discussion: Kubernetes-dashboard unreachable (sometimes)
Environment-Info:
Cloudstack 4.20.3.0
Hypervisor: VMware 7.0.3
Kubernetes-Iso: v1.36.0 from https://download.cloudstack.org/cks/setup-v1.36.0-x86_64.iso
Hi,
we're experimenting with cloudstack kubernetes service right now and in some cases the kubernetes-dashboard started locally with
kubectl --kubeconfig /path/to/kube.conf proxy
gives us a ServiceUnavailability-Message when called in browser
<img width="969" height="261" alt="09_04_51-000418" src="https://github.com/user-attachments/assets/11e46dfa-924e-408a-8a1d-5486a00d3112" />
So we started investigating and found out that the cloud-controller-manager often is in state CrashLoopBackOff and collect many restarts after a while:
```
kubectl --kubeconfig /mnt/d/Users/mwa/Downloads/vpc3-kube_kube.conf get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cloud-controller-manager-8648c8d5f-fmcwt 0/1 CrashLoopBackOff 899 (5m6s ago) 3d20h
kube-system coredns-589f44dc88-pwgz8 1/1 Running 0 3d20h
kube-system coredns-589f44dc88-x7kdm 1/1 Running 0 3d20h
kube-system etcd-vpc3-kube-control-19edf733c33 1/1 Running 0 3d20h
kube-system kube-apiserver-vpc3-kube-control-19edf733c33 1/1 Running 0 3d20h
kube-system kube-controller-manager-vpc3-kube-control-19edf733c33 1/1 Running 2 (26h ago) 3d20h
kube-system kube-proxy-6sv67 1/1 Running 0 3d20h
kube-system kube-proxy-srm5w 1/1 Running 0 3d20h
kube-system kube-scheduler-vpc3-kube-control-19edf733c33 1/1 Running 2 (26h ago) 3d20h
kube-system weave-net-d8zc8 2/2 Running 0 3d20h
kube-system weave-net-ttdnl 2/2 Running 1 (3d20h ago) 3d20h
kubernetes-dashboard dashboard-metrics-scraper-6c78786c6f-vvbkx 1/1 Running 0 3d20h
kubernetes-dashboard kubernetes-dashboard-57db5448b5-smgln 1/1 Running 0 3d20h
```
The log shows us a timeout trying to reach the management-server (192.168.0.87)
```
I0623 07:03:27.461933 1 serving.go:348] Generated self-signed cert in-memory
W0623 07:03:40.762340 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
F0623 07:04:10.764367 1 main.go:74] Cloud provider could not be initialized: could not init cloud provider "external-cloudstack": Get "https://cloudportal0.test.c.fraunhofer.de/clien
t/api/?apiKey=0s-l6KLik5QE7lTSd6NzlWEI5k0PSlOuFRvHREMILPg2Pm9R-_Nu7KPcUd2UriWb5lEqAA7SVQbJnJfrmjo6aA&command=listManagementServersMetrics&expires=2026-06-23T07%3A18%3A40Z&response=json&sig
natureversion=3&signature=R7yOS7KhVxOowH2g8rW4U2J753Y%3D": dial tcp 192.168.0.87:443: i/o timeout
```
This brought us to among other things to https://github.com/apache/cloudstack/discussions/12037 and https://github.com/apache/cloudstack/discussions/10258 and https://github.com/apache/cloudstack/discussions/9267
Leading us to the questions:
- Is the impossible callback to our management-server the root cause for this? (we had some dashboards running in some tests before although we used a completely wrong endpoint.url in our globals because we forgot about this setting in some earlier stadium).
- Will deployed applications be reachable at all? (we will test this parallel to this posting)
Since we're consider the VR-Networks as insecure in general and the cloudstack-ui is an internal-only resource would the reverse-proxy + firewall solution mentioned in the other discussions be advised for our scenario as well?
GitHub link: https://github.com/apache/cloudstack/discussions/13473
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Kubernetes-dashboard unreachable (sometimes) [cloudstack]
Posted by "chunkyen (via GitHub)" <gi...@apache.org>.
GitHub user chunkyen added a comment to the discussion: Kubernetes-dashboard unreachable (sometimes)
I would say that the crash is definitely caused by the inability for the CCM to reach the management server endpoint. I have a working kube 1.35 cluster with a fucntional CCM, but as soon as i stop the management server, I observe this in the CCM pod
<img width="874" height="56" alt="image" src="https://github.com/user-attachments/assets/1c04ffbf-72b0-4ab0-a603-1dc3fcf7ee3b" />
Note: It is a bit concerning that the api key is logged.
As soon as the management server is back up, the CCM pod started sucessfully
<img width="908" height="308" alt="image" src="https://github.com/user-attachments/assets/ed059772-23e0-41e9-8408-410e96526df8" />
GitHub link: https://github.com/apache/cloudstack/discussions/13473#discussioncomment-17414416
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Kubernetes-dashboard unreachable (sometimes) [cloudstack]
Posted by "chunkyen (via GitHub)" <gi...@apache.org>.
GitHub user chunkyen edited a comment on the discussion: Kubernetes-dashboard unreachable (sometimes)
I would say that the crash is definitely caused by the inability for the CCM to reach the management server endpoint. I have a working kube 1.35 cluster with a fucntional CCM, but as soon as i stop the management server, I observe this in the CCM pod
<img width="874" height="56" alt="image" src="https://github.com/user-attachments/assets/1c04ffbf-72b0-4ab0-a603-1dc3fcf7ee3b" />
As soon as the management server is back up, the CCM pod started sucessfully
<img width="908" height="308" alt="image" src="https://github.com/user-attachments/assets/ed059772-23e0-41e9-8408-410e96526df8" />
GitHub link: https://github.com/apache/cloudstack/discussions/13473#discussioncomment-17414416
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org