You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Ranjit Jadhav <ra...@gmail.com> on 2022/11/03 18:14:56 UTC
Issues with Kubernetes
Hello Folks,
We have an HA cloudstack setup but when we are creating a K8S cluster, the
cluster is not getting created, I can see that I'm able to console the
control node, but when I console the worker node, it says,
'Access is denied for the console session. Please close the window and
retry again.
I wasn't able to find any particular logs that could give me a clear
insight into the problem. However, I could find the find below logs that I
think could be the problem.
-------------------------------------------
Waiting for Kubernetes cluster : voyager control node VMs to be accessible
-------------------------------------------
Unable to find matched VM in CloudStack DB. name: i-2-11-VM
-------------------------------------------
Regards,
Ranjit
Re: Issues with Kubernetes
Posted by Jayanth Reddy <ja...@gmail.com>.
Hi,
Please list the below
1. CloudStack Version
2. Network (Isolated?)
Additionally, please ensure your management server(s) can reach the k8s
control plane nodes described at
https://github.com/apache/cloudstack/issues/4146#issuecomment-720327587
Thanks
On Mon, Nov 7, 2022 at 8:06 PM Ranjit Jadhav <ra...@gmail.com>
wrote:
> We are seeing following error in logs
>
> 2022-11-07 13:51:09,064 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-36:ctx-eae067a5 job-1464 ctx-03d611b7) (logid:42f2c60e)
> Seq 3-478788935384826404: Received: { Ans: , MgmtId: 112810681648291, via:
> 3(comp3), Ver: v1, Flags: 10, { Answer } }
> 2022-11-07 13:51:09,076 INFO [c.c.k.c.a.KubernetesClusterActionWorker]
> (API-Job-Executor-36:ctx-eae067a5 job-1464 ctx-03d611b7) (logid:42f2c60e)
> Detached Kubernetes binaries from VM : test-node-184522598f7 in the
> Kubernetes cluster : test
> 2022-11-07 13:51:09,077 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Complete async
> job-1464, jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> to setup Kubernetes cluster : test in usable state as unable to access
> control node VMs of the cluster"}
> 2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Publish async
> job-1464 complete on message bus
> 2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Wake up jobs
> related to job-1464
> 2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Update db
> status for job-1464
> 2022-11-07 13:51:09,079 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Wake up jobs
> joined with job-1464 and disjoin all subjobs created from job- 1464
> 2022-11-07 13:51:09,083 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Done executing
>
> org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd
> for job-1464
> --
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-11-07 14:02:21,654 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Complete async
> job-1505, jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"ip
> address is used for source nat purposes and can not be disassociated."}
> 2022-11-07 14:02:21,655 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-53:ctx-a2774db9 job-1506 ctx-60050a4c) (logid:8a26d8cc)
> Releasing 0 firewall rules for ip id=236
> 2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Publish async
> job-1505 complete on message bus
> 2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Wake up jobs
> related to job-1505
> 2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Update db
> status for job-1505
> 2022-11-07 14:02:21,656 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-53:ctx-a2774db9 job-1506 ctx-60050a4c) (logid:8a26d8cc)
> There are no firewall rules to apply
>
>
> Any guidelines will be helpful
>
> Regards,
> Ranjit
>
> On Thu, 3 Nov 2022 at 23:44, Ranjit Jadhav <ra...@gmail.com>
> wrote:
>
> > Hello Folks,
> >
> > We have an HA cloudstack setup but when we are creating a K8S cluster,
> the
> > cluster is not getting created, I can see that I'm able to console the
> > control node, but when I console the worker node, it says,
> >
> > 'Access is denied for the console session. Please close the window and
> > retry again.
> >
> > I wasn't able to find any particular logs that could give me a clear
> > insight into the problem. However, I could find the find below logs that
> I
> > think could be the problem.
> >
> > -------------------------------------------
> > Waiting for Kubernetes cluster : voyager control node VMs to be
> accessible
> > -------------------------------------------
> > Unable to find matched VM in CloudStack DB. name: i-2-11-VM
> > -------------------------------------------
> >
> > Regards,
> > Ranjit
> >
>
Re: Issues with Kubernetes
Posted by Jayanth Reddy <ja...@gmail.com>.
Hi,
Please list the below
1. CloudStack Version
2. Network (Isolated?)
Additionally, please ensure your management server(s) can reach the k8s
control plane nodes described at
https://github.com/apache/cloudstack/issues/4146#issuecomment-720327587
Thanks
On Mon, Nov 7, 2022 at 8:06 PM Ranjit Jadhav <ra...@gmail.com>
wrote:
> We are seeing following error in logs
>
> 2022-11-07 13:51:09,064 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-36:ctx-eae067a5 job-1464 ctx-03d611b7) (logid:42f2c60e)
> Seq 3-478788935384826404: Received: { Ans: , MgmtId: 112810681648291, via:
> 3(comp3), Ver: v1, Flags: 10, { Answer } }
> 2022-11-07 13:51:09,076 INFO [c.c.k.c.a.KubernetesClusterActionWorker]
> (API-Job-Executor-36:ctx-eae067a5 job-1464 ctx-03d611b7) (logid:42f2c60e)
> Detached Kubernetes binaries from VM : test-node-184522598f7 in the
> Kubernetes cluster : test
> 2022-11-07 13:51:09,077 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Complete async
> job-1464, jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> to setup Kubernetes cluster : test in usable state as unable to access
> control node VMs of the cluster"}
> 2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Publish async
> job-1464 complete on message bus
> 2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Wake up jobs
> related to job-1464
> 2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Update db
> status for job-1464
> 2022-11-07 13:51:09,079 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Wake up jobs
> joined with job-1464 and disjoin all subjobs created from job- 1464
> 2022-11-07 13:51:09,083 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Done executing
>
> org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd
> for job-1464
> --
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-11-07 14:02:21,654 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Complete async
> job-1505, jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"ip
> address is used for source nat purposes and can not be disassociated."}
> 2022-11-07 14:02:21,655 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-53:ctx-a2774db9 job-1506 ctx-60050a4c) (logid:8a26d8cc)
> Releasing 0 firewall rules for ip id=236
> 2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Publish async
> job-1505 complete on message bus
> 2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Wake up jobs
> related to job-1505
> 2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Update db
> status for job-1505
> 2022-11-07 14:02:21,656 DEBUG [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-53:ctx-a2774db9 job-1506 ctx-60050a4c) (logid:8a26d8cc)
> There are no firewall rules to apply
>
>
> Any guidelines will be helpful
>
> Regards,
> Ranjit
>
> On Thu, 3 Nov 2022 at 23:44, Ranjit Jadhav <ra...@gmail.com>
> wrote:
>
> > Hello Folks,
> >
> > We have an HA cloudstack setup but when we are creating a K8S cluster,
> the
> > cluster is not getting created, I can see that I'm able to console the
> > control node, but when I console the worker node, it says,
> >
> > 'Access is denied for the console session. Please close the window and
> > retry again.
> >
> > I wasn't able to find any particular logs that could give me a clear
> > insight into the problem. However, I could find the find below logs that
> I
> > think could be the problem.
> >
> > -------------------------------------------
> > Waiting for Kubernetes cluster : voyager control node VMs to be
> accessible
> > -------------------------------------------
> > Unable to find matched VM in CloudStack DB. name: i-2-11-VM
> > -------------------------------------------
> >
> > Regards,
> > Ranjit
> >
>
Re: Issues with Kubernetes
Posted by Ranjit Jadhav <ra...@gmail.com>.
We are seeing following error in logs
2022-11-07 13:51:09,064 DEBUG [c.c.a.t.Request]
(API-Job-Executor-36:ctx-eae067a5 job-1464 ctx-03d611b7) (logid:42f2c60e)
Seq 3-478788935384826404: Received: { Ans: , MgmtId: 112810681648291, via:
3(comp3), Ver: v1, Flags: 10, { Answer } }
2022-11-07 13:51:09,076 INFO [c.c.k.c.a.KubernetesClusterActionWorker]
(API-Job-Executor-36:ctx-eae067a5 job-1464 ctx-03d611b7) (logid:42f2c60e)
Detached Kubernetes binaries from VM : test-node-184522598f7 in the
Kubernetes cluster : test
2022-11-07 13:51:09,077 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Complete async
job-1464, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to setup Kubernetes cluster : test in usable state as unable to access
control node VMs of the cluster"}
2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Publish async
job-1464 complete on message bus
2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Wake up jobs
related to job-1464
2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Update db
status for job-1464
2022-11-07 13:51:09,079 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Wake up jobs
joined with job-1464 and disjoin all subjobs created from job- 1464
2022-11-07 13:51:09,083 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Done executing
org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd
for job-1464
--
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-11-07 14:02:21,654 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Complete async
job-1505, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"ip
address is used for source nat purposes and can not be disassociated."}
2022-11-07 14:02:21,655 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-53:ctx-a2774db9 job-1506 ctx-60050a4c) (logid:8a26d8cc)
Releasing 0 firewall rules for ip id=236
2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Publish async
job-1505 complete on message bus
2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Wake up jobs
related to job-1505
2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Update db
status for job-1505
2022-11-07 14:02:21,656 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-53:ctx-a2774db9 job-1506 ctx-60050a4c) (logid:8a26d8cc)
There are no firewall rules to apply
Any guidelines will be helpful
Regards,
Ranjit
On Thu, 3 Nov 2022 at 23:44, Ranjit Jadhav <ra...@gmail.com> wrote:
> Hello Folks,
>
> We have an HA cloudstack setup but when we are creating a K8S cluster, the
> cluster is not getting created, I can see that I'm able to console the
> control node, but when I console the worker node, it says,
>
> 'Access is denied for the console session. Please close the window and
> retry again.
>
> I wasn't able to find any particular logs that could give me a clear
> insight into the problem. However, I could find the find below logs that I
> think could be the problem.
>
> -------------------------------------------
> Waiting for Kubernetes cluster : voyager control node VMs to be accessible
> -------------------------------------------
> Unable to find matched VM in CloudStack DB. name: i-2-11-VM
> -------------------------------------------
>
> Regards,
> Ranjit
>
Re: Issues with Kubernetes
Posted by Ranjit Jadhav <ra...@gmail.com>.
We are seeing following error in logs
2022-11-07 13:51:09,064 DEBUG [c.c.a.t.Request]
(API-Job-Executor-36:ctx-eae067a5 job-1464 ctx-03d611b7) (logid:42f2c60e)
Seq 3-478788935384826404: Received: { Ans: , MgmtId: 112810681648291, via:
3(comp3), Ver: v1, Flags: 10, { Answer } }
2022-11-07 13:51:09,076 INFO [c.c.k.c.a.KubernetesClusterActionWorker]
(API-Job-Executor-36:ctx-eae067a5 job-1464 ctx-03d611b7) (logid:42f2c60e)
Detached Kubernetes binaries from VM : test-node-184522598f7 in the
Kubernetes cluster : test
2022-11-07 13:51:09,077 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Complete async
job-1464, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to setup Kubernetes cluster : test in usable state as unable to access
control node VMs of the cluster"}
2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Publish async
job-1464 complete on message bus
2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Wake up jobs
related to job-1464
2022-11-07 13:51:09,078 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Update db
status for job-1464
2022-11-07 13:51:09,079 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Wake up jobs
joined with job-1464 and disjoin all subjobs created from job- 1464
2022-11-07 13:51:09,083 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-36:ctx-eae067a5 job-1464) (logid:42f2c60e) Done executing
org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd
for job-1464
--
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-11-07 14:02:21,654 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Complete async
job-1505, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"ip
address is used for source nat purposes and can not be disassociated."}
2022-11-07 14:02:21,655 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-53:ctx-a2774db9 job-1506 ctx-60050a4c) (logid:8a26d8cc)
Releasing 0 firewall rules for ip id=236
2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Publish async
job-1505 complete on message bus
2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Wake up jobs
related to job-1505
2022-11-07 14:02:21,655 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-52:ctx-ad3791c7 job-1505) (logid:671c031b) Update db
status for job-1505
2022-11-07 14:02:21,656 DEBUG [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-53:ctx-a2774db9 job-1506 ctx-60050a4c) (logid:8a26d8cc)
There are no firewall rules to apply
Any guidelines will be helpful
Regards,
Ranjit
On Thu, 3 Nov 2022 at 23:44, Ranjit Jadhav <ra...@gmail.com> wrote:
> Hello Folks,
>
> We have an HA cloudstack setup but when we are creating a K8S cluster, the
> cluster is not getting created, I can see that I'm able to console the
> control node, but when I console the worker node, it says,
>
> 'Access is denied for the console session. Please close the window and
> retry again.
>
> I wasn't able to find any particular logs that could give me a clear
> insight into the problem. However, I could find the find below logs that I
> think could be the problem.
>
> -------------------------------------------
> Waiting for Kubernetes cluster : voyager control node VMs to be accessible
> -------------------------------------------
> Unable to find matched VM in CloudStack DB. name: i-2-11-VM
> -------------------------------------------
>
> Regards,
> Ranjit
>