You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Stanley Burkee <st...@gmail.com> on 2023/05/02 13:23:00 UTC
Problem with K8 cluster
Hi guys,
We are trying to create a Kubernetes cluster v.1.26 in Cloudstack 4.18. The
controller & worker nodes are provisioned and show in a running state but
Kubernetes Cluster is stuck in the starting state.
Thanks a lot in advance.
Regards
Stanley
Re: Problem with K8 cluster
Posted by Stephan Bienek <st...@bienek.org>.
Hi Stanley, Hi Wei,
thanks for your hints.
Indeed i just saw in CS 4.18, containerd >1.6 is already used.
So my issue is related to CS 4.17 (and its systemvm probably?)
Building an up to date kubernetes iso worked like a charm, but i am not sure if within the kubernetes iso i can instruct the kubernetes setup process to deploy an up to date containerd.
Anyways, after a CS update to 4.18 and its systemvm it should be solved anyways.
Best regards,
Stephan
> Wei ZHOU <us...@gmail.com> hat am 24.05.2023 20:44 CEST geschrieben:
>
>
> Hi Stephan,
>
> Which ISO do you use ? There are some ISOs on
> https://download.cloudstack.org/cks/ , but they are out-of-date.
> You can use create-kubernetes-binaries-iso.sh to create your own ISO with
> specific cni/cri/weave/dashboard versions.
>
>
> -Wei
>
>
> On Wed, 24 May 2023 at 18:05, Stephan Bienek <st...@bienek.org> wrote:
>
> > Hi Stanley,
> >
> > i finally found a hint, that for k8s v1.26 you need containerd >=1.6
> >
> > But the containerd version installed is < 1.6
> >
> > I made the "kubeadm init --token ..." command from the script
> > /opt/bin/deploy-kube-system working after updating containerd on the
> > controller node to >1.6
> >
> > echo "deb [arch=$(dpkg --print-architecture)
> > signed-by=/etc/apt/keyrings/docker.gpg]
> > https://download.docker.com/linux/debian $(lsb_release -cs) stable" |
> > sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
> > apt-get update
> > apt-get remove containerd runc
> > apt-get install containerd.io
> > containerd config default | tee /etc/containerd/config.toml
> > sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/'
> > /etc/containerd/config.toml
> > systemctl restart containerd
> >
> > bash -e /opt/bin/deploy-kube-system
> >
> > I'm now trying to find out which component is installing containerd and
> > where it's possible to adjust that.
> >
> > Best regards,
> > Stephan
> >
> > > Stanley Burkee <st...@gmail.com> hat am 04.05.2023 09:29 CEST
> > geschrieben:
> > >
> > >
> > > Hi Stephan,
> > >
> > > Thanks for your response.
> > >
> > > Please update if you were able to find any solution to the issue.
> > >
> > > Your help is much appreciated.
> > >
> > > Thanks
> > >
> > > Stanley
> > >
> > > On Tue, May 2, 2023 at 7:16 PM Stephan Bienek <st...@bienek.org>
> > > wrote:
> > >
> > > > Hi Stanley,
> > > >
> > > > i probably ran into the same issue on Cloudstack 4.17.0.1 and was not
> > sure
> > > > if it's an issue not beeing in 4.18 yet.
> > > >
> > > > The instances are up and running, but the kubebernetes cluster
> > deployment
> > > > issued on the control node via kubeadm is failing.
> > > >
> > > > When logging in to the control node and checking the kubeadm logs or
> > > > manually executing the bash script, which is executing kubeadm to
> > deploy
> > > > the cluster you will probably find the error
> > > >
> > > > CRI v1 runtime API is not implemented for endpoint
> > > > "unix:///run/containerd/containerd.sock": rpc error: code =
> > Unimplemented
> > > > desc = unknown service runtime.v1.RuntimeService
> > > >
> > > > and/or
> > > > [ERROR CRI]: container runtime is not running
> > > >
> > > > What i tried unsuccessful so far
> > > > - editing /etc/containerd/config.toml and adding a few tipps from
> > google
> > > > searches here and there
> > > > - deleting /etc/containerd/config.toml
> > > > - installing the more up to date version of containerd
> > > >
> > > > Unfortunately i couldn't find the time to fully troubleshoot the issue
> > yet
> > > > and went the easy way of using Kubernetes ISO v1.25 which works like a
> > > > charm.
> > > >
> > > > Maybe someone with more insight to it found an easy solution?
> > > >
> > > > Best regards,
> > > > Stephan
> > > >
> > > > > Stanley Burkee <st...@gmail.com> hat am 02.05.2023 15:23
> > CEST
> > > > geschrieben:
> > > > >
> > > > >
> > > > > Hi guys,
> > > > >
> > > > > We are trying to create a Kubernetes cluster v.1.26 in Cloudstack
> > 4.18.
> > > > The
> > > > > controller & worker nodes are provisioned and show in a running
> > state but
> > > > > Kubernetes Cluster is stuck in the starting state.
> > > > >
> > > > > Thanks a lot in advance.
> > > > >
> > > > >
> > > > > Regards
> > > > >
> > > > > Stanley
> > > >
> >
Re: Problem with K8 cluster
Posted by Wei ZHOU <us...@gmail.com>.
Hi Stephan,
Which ISO do you use ? There are some ISOs on
https://download.cloudstack.org/cks/ , but they are out-of-date.
You can use create-kubernetes-binaries-iso.sh to create your own ISO with
specific cni/cri/weave/dashboard versions.
-Wei
On Wed, 24 May 2023 at 18:05, Stephan Bienek <st...@bienek.org> wrote:
> Hi Stanley,
>
> i finally found a hint, that for k8s v1.26 you need containerd >=1.6
>
> But the containerd version installed is < 1.6
>
> I made the "kubeadm init --token ..." command from the script
> /opt/bin/deploy-kube-system working after updating containerd on the
> controller node to >1.6
>
> echo "deb [arch=$(dpkg --print-architecture)
> signed-by=/etc/apt/keyrings/docker.gpg]
> https://download.docker.com/linux/debian $(lsb_release -cs) stable" |
> sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
> apt-get update
> apt-get remove containerd runc
> apt-get install containerd.io
> containerd config default | tee /etc/containerd/config.toml
> sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/'
> /etc/containerd/config.toml
> systemctl restart containerd
>
> bash -e /opt/bin/deploy-kube-system
>
> I'm now trying to find out which component is installing containerd and
> where it's possible to adjust that.
>
> Best regards,
> Stephan
>
> > Stanley Burkee <st...@gmail.com> hat am 04.05.2023 09:29 CEST
> geschrieben:
> >
> >
> > Hi Stephan,
> >
> > Thanks for your response.
> >
> > Please update if you were able to find any solution to the issue.
> >
> > Your help is much appreciated.
> >
> > Thanks
> >
> > Stanley
> >
> > On Tue, May 2, 2023 at 7:16 PM Stephan Bienek <st...@bienek.org>
> > wrote:
> >
> > > Hi Stanley,
> > >
> > > i probably ran into the same issue on Cloudstack 4.17.0.1 and was not
> sure
> > > if it's an issue not beeing in 4.18 yet.
> > >
> > > The instances are up and running, but the kubebernetes cluster
> deployment
> > > issued on the control node via kubeadm is failing.
> > >
> > > When logging in to the control node and checking the kubeadm logs or
> > > manually executing the bash script, which is executing kubeadm to
> deploy
> > > the cluster you will probably find the error
> > >
> > > CRI v1 runtime API is not implemented for endpoint
> > > "unix:///run/containerd/containerd.sock": rpc error: code =
> Unimplemented
> > > desc = unknown service runtime.v1.RuntimeService
> > >
> > > and/or
> > > [ERROR CRI]: container runtime is not running
> > >
> > > What i tried unsuccessful so far
> > > - editing /etc/containerd/config.toml and adding a few tipps from
> google
> > > searches here and there
> > > - deleting /etc/containerd/config.toml
> > > - installing the more up to date version of containerd
> > >
> > > Unfortunately i couldn't find the time to fully troubleshoot the issue
> yet
> > > and went the easy way of using Kubernetes ISO v1.25 which works like a
> > > charm.
> > >
> > > Maybe someone with more insight to it found an easy solution?
> > >
> > > Best regards,
> > > Stephan
> > >
> > > > Stanley Burkee <st...@gmail.com> hat am 02.05.2023 15:23
> CEST
> > > geschrieben:
> > > >
> > > >
> > > > Hi guys,
> > > >
> > > > We are trying to create a Kubernetes cluster v.1.26 in Cloudstack
> 4.18.
> > > The
> > > > controller & worker nodes are provisioned and show in a running
> state but
> > > > Kubernetes Cluster is stuck in the starting state.
> > > >
> > > > Thanks a lot in advance.
> > > >
> > > >
> > > > Regards
> > > >
> > > > Stanley
> > >
>
Re: Problem with K8 cluster
Posted by Stanley Burkee <st...@gmail.com>.
Hi Stephan,
Thanks for the update.
I was able to run kubernetes v 1.26 on cloudstack 4.18. The dashboard service is by default running on the internal IP. You need to configure the node port for the dashboard service in order to access it from the outside envoirnment.
Once the node port is configured, you need to create corresponding port forwarding rule & firewall rule from kubernetes provisioned cluster.
After completing the above steps you will be able to get kubernetes dashboard.
Thanks
Best Regards
Stanley
Sent from Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Stephan Bienek <st...@bienek.org>
Sent: Wednesday, May 24, 2023 9:35:22 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Problem with K8 cluster
Hi Stanley,
i finally found a hint, that for k8s v1.26 you need containerd >=1.6
But the containerd version installed is < 1.6
I made the "kubeadm init --token ..." command from the script /opt/bin/deploy-kube-system working after updating containerd on the controller node to >1.6
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get remove containerd runc
apt-get install containerd.io
containerd config default | tee /etc/containerd/config.toml
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/' /etc/containerd/config.toml
systemctl restart containerd
bash -e /opt/bin/deploy-kube-system
I'm now trying to find out which component is installing containerd and where it's possible to adjust that.
Best regards,
Stephan
> Stanley Burkee <st...@gmail.com> hat am 04.05.2023 09:29 CEST geschrieben:
>
>
> Hi Stephan,
>
> Thanks for your response.
>
> Please update if you were able to find any solution to the issue.
>
> Your help is much appreciated.
>
> Thanks
>
> Stanley
>
> On Tue, May 2, 2023 at 7:16 PM Stephan Bienek <st...@bienek.org>
> wrote:
>
> > Hi Stanley,
> >
> > i probably ran into the same issue on Cloudstack 4.17.0.1 and was not sure
> > if it's an issue not beeing in 4.18 yet.
> >
> > The instances are up and running, but the kubebernetes cluster deployment
> > issued on the control node via kubeadm is failing.
> >
> > When logging in to the control node and checking the kubeadm logs or
> > manually executing the bash script, which is executing kubeadm to deploy
> > the cluster you will probably find the error
> >
> > CRI v1 runtime API is not implemented for endpoint
> > "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented
> > desc = unknown service runtime.v1.RuntimeService
> >
> > and/or
> > [ERROR CRI]: container runtime is not running
> >
> > What i tried unsuccessful so far
> > - editing /etc/containerd/config.toml and adding a few tipps from google
> > searches here and there
> > - deleting /etc/containerd/config.toml
> > - installing the more up to date version of containerd
> >
> > Unfortunately i couldn't find the time to fully troubleshoot the issue yet
> > and went the easy way of using Kubernetes ISO v1.25 which works like a
> > charm.
> >
> > Maybe someone with more insight to it found an easy solution?
> >
> > Best regards,
> > Stephan
> >
> > > Stanley Burkee <st...@gmail.com> hat am 02.05.2023 15:23 CEST
> > geschrieben:
> > >
> > >
> > > Hi guys,
> > >
> > > We are trying to create a Kubernetes cluster v.1.26 in Cloudstack 4.18.
> > The
> > > controller & worker nodes are provisioned and show in a running state but
> > > Kubernetes Cluster is stuck in the starting state.
> > >
> > > Thanks a lot in advance.
> > >
> > >
> > > Regards
> > >
> > > Stanley
> >
Re: Problem with K8 cluster
Posted by Stephan Bienek <st...@bienek.org>.
Hi Stanley,
i finally found a hint, that for k8s v1.26 you need containerd >=1.6
But the containerd version installed is < 1.6
I made the "kubeadm init --token ..." command from the script /opt/bin/deploy-kube-system working after updating containerd on the controller node to >1.6
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get remove containerd runc
apt-get install containerd.io
containerd config default | tee /etc/containerd/config.toml
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/' /etc/containerd/config.toml
systemctl restart containerd
bash -e /opt/bin/deploy-kube-system
I'm now trying to find out which component is installing containerd and where it's possible to adjust that.
Best regards,
Stephan
> Stanley Burkee <st...@gmail.com> hat am 04.05.2023 09:29 CEST geschrieben:
>
>
> Hi Stephan,
>
> Thanks for your response.
>
> Please update if you were able to find any solution to the issue.
>
> Your help is much appreciated.
>
> Thanks
>
> Stanley
>
> On Tue, May 2, 2023 at 7:16 PM Stephan Bienek <st...@bienek.org>
> wrote:
>
> > Hi Stanley,
> >
> > i probably ran into the same issue on Cloudstack 4.17.0.1 and was not sure
> > if it's an issue not beeing in 4.18 yet.
> >
> > The instances are up and running, but the kubebernetes cluster deployment
> > issued on the control node via kubeadm is failing.
> >
> > When logging in to the control node and checking the kubeadm logs or
> > manually executing the bash script, which is executing kubeadm to deploy
> > the cluster you will probably find the error
> >
> > CRI v1 runtime API is not implemented for endpoint
> > "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented
> > desc = unknown service runtime.v1.RuntimeService
> >
> > and/or
> > [ERROR CRI]: container runtime is not running
> >
> > What i tried unsuccessful so far
> > - editing /etc/containerd/config.toml and adding a few tipps from google
> > searches here and there
> > - deleting /etc/containerd/config.toml
> > - installing the more up to date version of containerd
> >
> > Unfortunately i couldn't find the time to fully troubleshoot the issue yet
> > and went the easy way of using Kubernetes ISO v1.25 which works like a
> > charm.
> >
> > Maybe someone with more insight to it found an easy solution?
> >
> > Best regards,
> > Stephan
> >
> > > Stanley Burkee <st...@gmail.com> hat am 02.05.2023 15:23 CEST
> > geschrieben:
> > >
> > >
> > > Hi guys,
> > >
> > > We are trying to create a Kubernetes cluster v.1.26 in Cloudstack 4.18.
> > The
> > > controller & worker nodes are provisioned and show in a running state but
> > > Kubernetes Cluster is stuck in the starting state.
> > >
> > > Thanks a lot in advance.
> > >
> > >
> > > Regards
> > >
> > > Stanley
> >
Re: Problem with K8 cluster
Posted by Stanley Burkee <st...@gmail.com>.
Hi Stephan,
Thanks for your response.
Please update if you were able to find any solution to the issue.
Your help is much appreciated.
Thanks
Stanley
On Tue, May 2, 2023 at 7:16 PM Stephan Bienek <st...@bienek.org>
wrote:
> Hi Stanley,
>
> i probably ran into the same issue on Cloudstack 4.17.0.1 and was not sure
> if it's an issue not beeing in 4.18 yet.
>
> The instances are up and running, but the kubebernetes cluster deployment
> issued on the control node via kubeadm is failing.
>
> When logging in to the control node and checking the kubeadm logs or
> manually executing the bash script, which is executing kubeadm to deploy
> the cluster you will probably find the error
>
> CRI v1 runtime API is not implemented for endpoint
> "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented
> desc = unknown service runtime.v1.RuntimeService
>
> and/or
> [ERROR CRI]: container runtime is not running
>
> What i tried unsuccessful so far
> - editing /etc/containerd/config.toml and adding a few tipps from google
> searches here and there
> - deleting /etc/containerd/config.toml
> - installing the more up to date version of containerd
>
> Unfortunately i couldn't find the time to fully troubleshoot the issue yet
> and went the easy way of using Kubernetes ISO v1.25 which works like a
> charm.
>
> Maybe someone with more insight to it found an easy solution?
>
> Best regards,
> Stephan
>
> > Stanley Burkee <st...@gmail.com> hat am 02.05.2023 15:23 CEST
> geschrieben:
> >
> >
> > Hi guys,
> >
> > We are trying to create a Kubernetes cluster v.1.26 in Cloudstack 4.18.
> The
> > controller & worker nodes are provisioned and show in a running state but
> > Kubernetes Cluster is stuck in the starting state.
> >
> > Thanks a lot in advance.
> >
> >
> > Regards
> >
> > Stanley
>
Re: Problem with K8 cluster
Posted by Stephan Bienek <st...@bienek.org>.
Hi Stanley,
i probably ran into the same issue on Cloudstack 4.17.0.1 and was not sure if it's an issue not beeing in 4.18 yet.
The instances are up and running, but the kubebernetes cluster deployment issued on the control node via kubeadm is failing.
When logging in to the control node and checking the kubeadm logs or manually executing the bash script, which is executing kubeadm to deploy the cluster you will probably find the error
CRI v1 runtime API is not implemented for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService
and/or
[ERROR CRI]: container runtime is not running
What i tried unsuccessful so far
- editing /etc/containerd/config.toml and adding a few tipps from google searches here and there
- deleting /etc/containerd/config.toml
- installing the more up to date version of containerd
Unfortunately i couldn't find the time to fully troubleshoot the issue yet and went the easy way of using Kubernetes ISO v1.25 which works like a charm.
Maybe someone with more insight to it found an easy solution?
Best regards,
Stephan
> Stanley Burkee <st...@gmail.com> hat am 02.05.2023 15:23 CEST geschrieben:
>
>
> Hi guys,
>
> We are trying to create a Kubernetes cluster v.1.26 in Cloudstack 4.18. The
> controller & worker nodes are provisioned and show in a running state but
> Kubernetes Cluster is stuck in the starting state.
>
> Thanks a lot in advance.
>
>
> Regards
>
> Stanley