You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "n4l5u0r (via GitHub)" <gi...@apache.org> on 2025/10/31 13:54:16 UTC
[D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]
GitHub user n4l5u0r created a discussion: Unable to have k8s Running in Cloudstack 4.20.0
Trying to deploy a k8s from Cloudstack CKS
based on
mgmt servers : 10.10.0.0/20 / 10.20.0.0/20
- mgmt-1 : 10.X0.0.3
- mgmt-2 : 10.X0.0.4
KVM servers: 10.10.0.0/20, 10.20.0.0/20, 10.40.0.0/20
- kvm-1 : 10.X0.0.6
- kvm-2 : 10.X0.0.7
When deploying the overall k8s everything starts, the instances controle + node, the VR responsible of the isolated network : 10.30.0.0/16
the endpoint is never set to `Running`, remains on `Starting` and ends in `Error`.
<img width="938" height="345" alt="image" src="https://github.com/user-attachments/assets/fdf19f14-8ecd-48ba-aca7-449cf91c3b6c" />
the `endpoint.url`is set to `http://10.10.0.3:8080/client/api`
the physical network topology is the following:
- Mgmt network : 10.10.0.0/20 : mgmt + kvm
- Storage network : 10.20.0.0/20 : mgmt + kvm
- Public network : 10.40.0.0/20 : kvm
- Guest network : 10.30.0.0/16
I don't know if the problem is coming from the fact that the Mgmt servers are not accessing the Public network, where the k8s network VR is exposing the k8s services, to check the availability of control service availability.
If I have to expose it what is the best practice on Cloudstack side to have working.
<img width="724" height="332" alt="image" src="https://github.com/user-attachments/assets/58c63969-15f7-4e54-b495-fbb0174d9e1c" />
<img width="1294" height="860" alt="image" src="https://github.com/user-attachments/assets/66859cb1-222f-401e-bfd1-45101900ef03" />
<img width="927" height="411" alt="image" src="https://github.com/user-attachments/assets/7c7f307a-d6c4-4727-bc3d-cb0589e9970e" />
when I request the k8s service from a kvm I have it working up:
```bash
# curl -sk https://10.40.0.12:6443/version
{
"major": "1",
"minor": "32",
"gitVersion": "v1.32.5",
"gitCommit": "9894294ef13a5b32803e3ca2c0d620a088cc84d1",
"gitTreeState": "clean",
"buildDate": "2025-05-15T09:05:02Z",
"goVersion": "go1.23.8",
"compiler": "gc",
"platform": "linux/amd64"
}```
GitHub link: https://github.com/apache/cloudstack/discussions/11951
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]
Posted by "n4l5u0r (via GitHub)" <gi...@apache.org>.
GitHub user n4l5u0r added a comment to the discussion: Unable to have k8s Running in Cloudstack 4.20.0
looks like having issues here :
```bash
cloud@test-k8S-control-19a3a5cfc67:~$ sudo -i
root@test-k8S-control-19a3a5cfc67:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
test-k8s-control-19a3a5cfc67 Ready control-plane 63m v1.32.5
root@test-k8S-control-19a3a5cfc67:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7bfdc5b57c-9vnph 0/1 ContainerCreating 0 56m
kube-system calico-node-2q75j 0/1 CrashLoopBackOff 11 (25m ago) 56m
kube-system coredns-668d6bf9bc-5v8n5 0/1 Pending 0 63m
kube-system coredns-668d6bf9bc-s8blq 0/1 Pending 0 63m
kube-system etcd-test-k8s-control-19a3a5cfc67 1/1 Running 0 64m
kube-system kube-apiserver-test-k8s-control-19a3a5cfc67 1/1 Running 0 64m
kube-system kube-controller-manager-test-k8s-control-19a3a5cfc67 1/1 Running 0 64m
kube-system kube-proxy-lkbjz 1/1 Running 0 63m
kube-system kube-scheduler-test-k8s-control-19a3a5cfc67 1/1 Running 0 64m
```
``` bash
root@test-k8S-control-19a3a5cfc67:~# kubectl get event -A
NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
kube-system 58m Warning FailedScheduling pod/calico-kube-controllers-7bfdc5b57c-9vnph 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system 58m Normal Scheduled pod/calico-kube-controllers-7bfdc5b57c-9vnph Successfully assigned kube-system/calico-kube-controllers-7bfdc5b57c-9vnph to test-k8s-control-19a3a5cfc67
kube-system 58m Normal SuccessfulCreate replicaset/calico-kube-controllers-7bfdc5b57c Created pod: calico-kube-controllers-7bfdc5b57c-9vnph
kube-system 58m Normal ScalingReplicaSet deployment/calico-kube-controllers Scaled up replica set calico-kube-controllers-7bfdc5b57c from 0 to 1
kube-system 58m Normal Scheduled pod/calico-node-2q75j Successfully assigned kube-system/calico-node-2q75j to test-k8s-control-19a3a5cfc67
kube-system 58m Normal Pulled pod/calico-node-2q75j Container image "docker.io/calico/cni:v3.30.0" already present on machine
kube-system 58m Normal Created pod/calico-node-2q75j Created container: upgrade-ipam
kube-system 58m Normal Started pod/calico-node-2q75j Started container upgrade-ipam
kube-system 58m Normal Pulled pod/calico-node-2q75j Container image "docker.io/calico/cni:v3.30.0" already present on machine
kube-system 58m Normal Created pod/calico-node-2q75j Created container: install-cni
kube-system 58m Normal Pulled pod/calico-node-2q75j Container image "docker.io/calico/node:v3.30.0" already present on machine
kube-system 58m Normal Started pod/calico-node-2q75j Started container mount-bpffs
kube-system 58m Normal Pulled pod/calico-node-2q75j Container image "docker.io/calico/node:v3.30.0" already present on machine
kube-system 58m Normal Created pod/calico-node-2q75j Created container: calico-node
kube-system 58m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-2q75j
kube-system 60m Warning FailedScheduling pod/coredns-668d6bf9bc-5v8n5 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system 58m Normal Scheduled pod/coredns-668d6bf9bc-5v8n5 Successfully assigned kube-system/coredns-668d6bf9bc-5v8n5 to test-k8s-control-19a3a5cfc67
kube-system 60m Warning FailedScheduling pod/coredns-668d6bf9bc-s8blq 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system 58m Normal Scheduled pod/coredns-668d6bf9bc-s8blq Successfully assigned kube-system/coredns-668d6bf9bc-s8blq to test-k8s-control-19a3a5cfc67
```
GitHub link: https://github.com/apache/cloudstack/discussions/11951#discussioncomment-14838974
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]
Posted by "Pearl1594 (via GitHub)" <gi...@apache.org>.
GitHub user Pearl1594 added a comment to the discussion: Unable to have k8s Running in Cloudstack 4.20.0
@n4l5u0r Can you try to SSH into the control node using:
ssh -i <SSH key / MS SSH key> -p 2222 cloud@10.40.0.12
And then execute the following:
```
sudo -i
kubectl get nodes
kubectl get po -A
```
GitHub link: https://github.com/apache/cloudstack/discussions/11951#discussioncomment-14838795
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]
Posted by "Pearl1594 (via GitHub)" <gi...@apache.org>.
GitHub user Pearl1594 added a comment to the discussion: Unable to have k8s Running in Cloudstack 4.20.0
Seems like the worker node did not come up.
Can you log into the worker node: ssh -i <SSH key / MS SSH key> -p 2223 cloud@10.40.0.12
And check if /var/log/daemon.log or /var/log/cloud-init-output.log or systemctl status deploy-kube-system indicate why the worker node failed to come up..
GitHub link: https://github.com/apache/cloudstack/discussions/11951#discussioncomment-14839136
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]
Posted by "n4l5u0r (via GitHub)" <gi...@apache.org>.
GitHub user n4l5u0r closed a discussion: Unable to have k8s Running in Cloudstack 4.20.0
Trying to deploy a k8s from Cloudstack CKS
based on
mgmt servers : 10.10.0.0/20 / 10.20.0.0/20
- mgmt-1 : 10.X0.0.3
- mgmt-2 : 10.X0.0.4
KVM servers: 10.10.0.0/20, 10.20.0.0/20, 10.40.0.0/20
- kvm-1 : 10.X0.0.6
- kvm-2 : 10.X0.0.7
When deploying the overall k8s everything starts, the instances controle + node, the VR responsible of the isolated network : 10.30.0.0/16
the endpoint is never set to `Running`, remains on `Starting` and ends in `Error`.
<img width="938" height="345" alt="image" src="https://github.com/user-attachments/assets/fdf19f14-8ecd-48ba-aca7-449cf91c3b6c" />
the `endpoint.url`is set to `http://10.10.0.3:8080/client/api`
the physical network topology is the following:
- Mgmt network : 10.10.0.0/20 : mgmt + kvm
- Storage network : 10.20.0.0/20 : mgmt + kvm
- Public network : 10.40.0.0/20 : kvm
- Guest network : 10.30.0.0/16
I don't know if the problem is coming from the fact that the Mgmt servers are not accessing the Public network, where the k8s network VR is exposing the k8s services, to check the availability of control service availability.
If I have to expose it what is the best practice on Cloudstack side to have working.
<img width="724" height="332" alt="image" src="https://github.com/user-attachments/assets/58c63969-15f7-4e54-b495-fbb0174d9e1c" />
<img width="1294" height="860" alt="image" src="https://github.com/user-attachments/assets/66859cb1-222f-401e-bfd1-45101900ef03" />
<img width="927" height="411" alt="image" src="https://github.com/user-attachments/assets/7c7f307a-d6c4-4727-bc3d-cb0589e9970e" />
when I request the k8s service from a kvm I have it working up:
```bash
# curl -sk https://10.40.0.12:6443/version
{
"major": "1",
"minor": "32",
"gitVersion": "v1.32.5",
"gitCommit": "9894294ef13a5b32803e3ca2c0d620a088cc84d1",
"gitTreeState": "clean",
"buildDate": "2025-05-15T09:05:02Z",
"goVersion": "go1.23.8",
"compiler": "gc",
"platform": "linux/amd64"
}```
GitHub link: https://github.com/apache/cloudstack/discussions/11951
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org