You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "n4l5u0r (via GitHub)" <gi...@apache.org> on 2025/10/31 13:54:16 UTC

[D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]

GitHub user n4l5u0r created a discussion: Unable to have k8s Running in Cloudstack 4.20.0

Trying to deploy a k8s from Cloudstack CKS
based on 

mgmt servers : 10.10.0.0/20 / 10.20.0.0/20
- mgmt-1 : 10.X0.0.3
- mgmt-2 : 10.X0.0.4

KVM servers: 10.10.0.0/20, 10.20.0.0/20, 10.40.0.0/20
 - kvm-1 : 10.X0.0.6
 - kvm-2 : 10.X0.0.7

When deploying the overall k8s everything starts, the instances controle + node, the VR responsible of the isolated network : 10.30.0.0/16

the endpoint is never set to `Running`, remains on `Starting` and ends in `Error`.

<img width="938" height="345" alt="image" src="https://github.com/user-attachments/assets/fdf19f14-8ecd-48ba-aca7-449cf91c3b6c" />

the `endpoint.url`is set to `http://10.10.0.3:8080/client/api`

the physical network topology is the following:
- Mgmt network : 10.10.0.0/20 : mgmt + kvm
- Storage network : 10.20.0.0/20 : mgmt + kvm
- Public network : 10.40.0.0/20 : kvm
- Guest network : 10.30.0.0/16 

I don't know if the problem is coming from the fact that the Mgmt servers are not accessing the Public network, where the k8s network VR is exposing the k8s services, to check the availability of control service availability.

If I have to expose it what is the best practice on Cloudstack side to have working. 

<img width="724" height="332" alt="image" src="https://github.com/user-attachments/assets/58c63969-15f7-4e54-b495-fbb0174d9e1c" />

<img width="1294" height="860" alt="image" src="https://github.com/user-attachments/assets/66859cb1-222f-401e-bfd1-45101900ef03" />

<img width="927" height="411" alt="image" src="https://github.com/user-attachments/assets/7c7f307a-d6c4-4727-bc3d-cb0589e9970e" />


when I request the k8s service from a kvm I have it working up:

```bash
# curl -sk https://10.40.0.12:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.5",
  "gitCommit": "9894294ef13a5b32803e3ca2c0d620a088cc84d1",
  "gitTreeState": "clean",
  "buildDate": "2025-05-15T09:05:02Z",
  "goVersion": "go1.23.8",
  "compiler": "gc",
  "platform": "linux/amd64"
}```


GitHub link: https://github.com/apache/cloudstack/discussions/11951

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]

Posted by "n4l5u0r (via GitHub)" <gi...@apache.org>.
GitHub user n4l5u0r added a comment to the discussion: Unable to have k8s Running in Cloudstack 4.20.0

looks like having issues here : 

```bash

cloud@test-k8S-control-19a3a5cfc67:~$ sudo -i
root@test-k8S-control-19a3a5cfc67:~# kubectl get nodes
NAME                           STATUS   ROLES           AGE   VERSION
test-k8s-control-19a3a5cfc67   Ready    control-plane   63m   v1.32.5
root@test-k8S-control-19a3a5cfc67:~# kubectl get pods -A
NAMESPACE     NAME                                                   READY   STATUS              RESTARTS       AGE
kube-system   calico-kube-controllers-7bfdc5b57c-9vnph               0/1     ContainerCreating   0              56m
kube-system   calico-node-2q75j                                      0/1     CrashLoopBackOff    11 (25m ago)   56m
kube-system   coredns-668d6bf9bc-5v8n5                               0/1     Pending             0              63m
kube-system   coredns-668d6bf9bc-s8blq                               0/1     Pending             0              63m
kube-system   etcd-test-k8s-control-19a3a5cfc67                      1/1     Running             0              64m
kube-system   kube-apiserver-test-k8s-control-19a3a5cfc67            1/1     Running             0              64m
kube-system   kube-controller-manager-test-k8s-control-19a3a5cfc67   1/1     Running             0              64m
kube-system   kube-proxy-lkbjz                                       1/1     Running             0              63m
kube-system   kube-scheduler-test-k8s-control-19a3a5cfc67            1/1     Running             0              64m

``` 

``` bash
root@test-k8S-control-19a3a5cfc67:~# kubectl get event -A
NAMESPACE     LAST SEEN   TYPE      REASON              OBJECT                                          MESSAGE
kube-system   58m         Warning   FailedScheduling    pod/calico-kube-controllers-7bfdc5b57c-9vnph    0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system   58m         Normal    Scheduled           pod/calico-kube-controllers-7bfdc5b57c-9vnph    Successfully assigned kube-system/calico-kube-controllers-7bfdc5b57c-9vnph to test-k8s-control-19a3a5cfc67
kube-system   58m         Normal    SuccessfulCreate    replicaset/calico-kube-controllers-7bfdc5b57c   Created pod: calico-kube-controllers-7bfdc5b57c-9vnph
kube-system   58m         Normal    ScalingReplicaSet   deployment/calico-kube-controllers              Scaled up replica set calico-kube-controllers-7bfdc5b57c from 0 to 1
kube-system   58m         Normal    Scheduled           pod/calico-node-2q75j                           Successfully assigned kube-system/calico-node-2q75j to test-k8s-control-19a3a5cfc67
kube-system   58m         Normal    Pulled              pod/calico-node-2q75j                           Container image "docker.io/calico/cni:v3.30.0" already present on machine
kube-system   58m         Normal    Created             pod/calico-node-2q75j                           Created container: upgrade-ipam
kube-system   58m         Normal    Started             pod/calico-node-2q75j                           Started container upgrade-ipam
kube-system   58m         Normal    Pulled              pod/calico-node-2q75j                           Container image "docker.io/calico/cni:v3.30.0" already present on machine
kube-system   58m         Normal    Created             pod/calico-node-2q75j                           Created container: install-cni
kube-system   58m         Normal    Pulled              pod/calico-node-2q75j                           Container image "docker.io/calico/node:v3.30.0" already present on machine
kube-system   58m         Normal    Started             pod/calico-node-2q75j                           Started container mount-bpffs
kube-system   58m         Normal    Pulled              pod/calico-node-2q75j                           Container image "docker.io/calico/node:v3.30.0" already present on machine
kube-system   58m         Normal    Created             pod/calico-node-2q75j                           Created container: calico-node
kube-system   58m         Normal    SuccessfulCreate    daemonset/calico-node                           Created pod: calico-node-2q75j
kube-system   60m         Warning   FailedScheduling    pod/coredns-668d6bf9bc-5v8n5                    0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system   58m         Normal    Scheduled           pod/coredns-668d6bf9bc-5v8n5                    Successfully assigned kube-system/coredns-668d6bf9bc-5v8n5 to test-k8s-control-19a3a5cfc67
kube-system   60m         Warning   FailedScheduling    pod/coredns-668d6bf9bc-s8blq                    0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
kube-system   58m         Normal    Scheduled           pod/coredns-668d6bf9bc-s8blq                    Successfully assigned kube-system/coredns-668d6bf9bc-s8blq to test-k8s-control-19a3a5cfc67

``` 


GitHub link: https://github.com/apache/cloudstack/discussions/11951#discussioncomment-14838974

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]

Posted by "Pearl1594 (via GitHub)" <gi...@apache.org>.
GitHub user Pearl1594 added a comment to the discussion: Unable to have k8s Running in Cloudstack 4.20.0

@n4l5u0r Can you try to SSH into the control node using:
ssh -i <SSH key / MS SSH key> -p 2222 cloud@10.40.0.12

And then execute the following:
```
sudo -i
kubectl get nodes
kubectl get po -A
```

GitHub link: https://github.com/apache/cloudstack/discussions/11951#discussioncomment-14838795

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]

Posted by "Pearl1594 (via GitHub)" <gi...@apache.org>.
GitHub user Pearl1594 added a comment to the discussion: Unable to have k8s Running in Cloudstack 4.20.0

Seems like the worker node did not come up.
Can you log into the worker node: ssh -i <SSH key / MS SSH key> -p 2223 cloud@10.40.0.12

And check if /var/log/daemon.log or /var/log/cloud-init-output.log or systemctl status deploy-kube-system indicate why the worker node failed to come up..


GitHub link: https://github.com/apache/cloudstack/discussions/11951#discussioncomment-14839136

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org


Re: [D] Unable to have k8s Running in Cloudstack 4.20.0 [cloudstack]

Posted by "n4l5u0r (via GitHub)" <gi...@apache.org>.
GitHub user n4l5u0r closed a discussion: Unable to have k8s Running in Cloudstack 4.20.0

Trying to deploy a k8s from Cloudstack CKS
based on 

mgmt servers : 10.10.0.0/20 / 10.20.0.0/20
- mgmt-1 : 10.X0.0.3
- mgmt-2 : 10.X0.0.4

KVM servers: 10.10.0.0/20, 10.20.0.0/20, 10.40.0.0/20
 - kvm-1 : 10.X0.0.6
 - kvm-2 : 10.X0.0.7

When deploying the overall k8s everything starts, the instances controle + node, the VR responsible of the isolated network : 10.30.0.0/16

the endpoint is never set to `Running`, remains on `Starting` and ends in `Error`.

<img width="938" height="345" alt="image" src="https://github.com/user-attachments/assets/fdf19f14-8ecd-48ba-aca7-449cf91c3b6c" />

the `endpoint.url`is set to `http://10.10.0.3:8080/client/api`

the physical network topology is the following:
- Mgmt network : 10.10.0.0/20 : mgmt + kvm
- Storage network : 10.20.0.0/20 : mgmt + kvm
- Public network : 10.40.0.0/20 : kvm
- Guest network : 10.30.0.0/16 

I don't know if the problem is coming from the fact that the Mgmt servers are not accessing the Public network, where the k8s network VR is exposing the k8s services, to check the availability of control service availability.

If I have to expose it what is the best practice on Cloudstack side to have working. 

<img width="724" height="332" alt="image" src="https://github.com/user-attachments/assets/58c63969-15f7-4e54-b495-fbb0174d9e1c" />

<img width="1294" height="860" alt="image" src="https://github.com/user-attachments/assets/66859cb1-222f-401e-bfd1-45101900ef03" />

<img width="927" height="411" alt="image" src="https://github.com/user-attachments/assets/7c7f307a-d6c4-4727-bc3d-cb0589e9970e" />


when I request the k8s service from a kvm I have it working up:

```bash
# curl -sk https://10.40.0.12:6443/version
{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.5",
  "gitCommit": "9894294ef13a5b32803e3ca2c0d620a088cc84d1",
  "gitTreeState": "clean",
  "buildDate": "2025-05-15T09:05:02Z",
  "goVersion": "go1.23.8",
  "compiler": "gc",
  "platform": "linux/amd64"
}```


GitHub link: https://github.com/apache/cloudstack/discussions/11951

----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org