You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by David Jumani <Da...@shapeblue.com> on 2022/08/01 03:32:05 UTC
Re: Issues with cloudstack-kubernetes-provider load balancer
Hi Kiran
I've discussed this issue with Pierre offline.
This happens when the names of the VMs in CloudStack do not match the node names in Kubernetes, likely an upper-lower case issue
A PR to fix it has been raised : https://github.com/apache/cloudstack-kubernetes-provider/pull/41
As a workaround, you can rename the VMs in CloudStack to exactly match the output of `kubectl get nodes`
________________________________
From: Kiran manohar Chavala <ki...@gmail.com>
Sent: Sunday, July 31, 2022 11:21 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Issues with cloudstack-kubernetes-provider load balancer
Hi Pierre Le Fevre
Facing a similar kind of issue when trying to acquire Loadbalncer Ip
Any workaround for the issue
❯ k describe svc kiran
Name: kiran
Namespace: default
Labels: run=nginx
Annotations: <none>
Selector: run=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.99.211.32
IPs: 10.99.211.32
Port: <unset> 80/TCP
TargetPort: 8000/TCP
NodePort: <unset> 31588/TCP
Endpoints: 10.44.0.4:8000,10.44.0.5:8000
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From
Message
---- ------ ---- ----
-------
Normal EnsuringLoadBalancer 61s (x2 over 2m31s) service-controller
Ensuring load balancer
Warning SyncLoadBalancerFailed 31s (x2 over 2m1s) service-controller
Error syncing load balancer: failed to ensure load balancer: error
retrieving load balancer rules: Get
http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA&command=listLoadBalancerRules&keyword=a94511f7024b24efdaae576625a6fe68&listall=true&response=json&signature=bpLHBJTA1LgLGV9DCevvKshu2Zg%3D:
dial tcp 10.102.192.220:8080: i/o timeout
Regards
Kiran
On Thu, Jun 16, 2022 at 8:33 PM Pierre Le Fevre <pi...@kth.se> wrote:
> Hi all,
>
> I have recently been experimenting with Kubernetes in CloudStack and came
> across a quite odd problem with the communication
> between cloudstack-kubernetes-provider and the CloudStack API.
>
> After some experimentation with the cloud-config api-url variable, it seems
> that connection should be happening. I tried using wget from within
> the pods with success.
> However, whenever services try to sync the load balancer to CloudStack, the
> error* Error syncing load balancer: failed to ensure load balancer: could
> not find network *shows up.
>
> Taking a look at the logs of *cloud-controller-manager*, the same three
> rows come up repeatedly:
>
> I0616 14:02:26.189085 1 event.go:278]
> Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
> UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> ResourceVersion:"1776", FieldPath:""}): type: 'Normal' reason:
> 'EnsuringLoadBalancer' Ensuring load balancer
>
> E0616 14:02:26.273801 1 controller.go:244] error processing service
> default/balls (will retry): failed to ensure load balancer: could not find
> network
>
> I0616 14:02:26.274430 1 event.go:278]
> Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
> UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> ResourceVersion:"1776", FieldPath:""}): type: 'Warning' reason:
> 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
> balancer: could not find network
>
>
> When watching the logs of our management server, the request
> *listNetworks *appears,
> coming from the kubernetes IP. It does, however, include an empty *id
> *parameter ,
> which in turn returns the error:
>
> {
> "listnetworksresponse": {
> "uuidList": [],
> "errorcode": 431,
> "cserrorcode": 9999,
> "errortext": "Unable to execute API command listnetworks due to
> invalid value. Invalid parameter id value= due to incorrect long value
> format, or entity does not exist or due to incorrect parameter annotation
> for the field in api cmd class."
> }
> }
>
> The request: http://*domain*/client/api?apiKey=*key*&command=listNetworks&
> *id=*&response=json&signature=*signature*
>
> Looking at the code of cloudstack-kubernetes-provider
> (cloudstack_loadbalancer.go:417), it seems like the error should return
> *... **could not find network [network ID]*, but in our case, no ID shows
> up (it could be nil?).
> It appears that cs.verifyHosts (cloudstack_loadbalancer.go:100) is failing
> to update lb.networkID, which cascades into this error.
>
> Seeing as it shows up even after creating different clusters, could this
> still be a configuration issue, or is it a known bug in CKS?
>
> Thanks in advance,
> Pierre Le Fevre
>
Re: Issues with cloudstack-kubernetes-provider load balancer
Posted by Kiran manohar Chavala <ki...@gmail.com>.
Hi David
I think the names are fine ,Please find the kubectl output and screenshot
attached
-------
❯ kubectl get nodes
NAME STATUS ROLES AGE
VERSION
kiran-kube-cluster-control-18254bced26 Ready control-plane 13h
v1.24.0
kiran-kube-cluster-node-18254bf3fef Ready <none> 13h
v1.24.0
-------
Also when I give kubectl get svc is the external-ip expected to appear?
The load balancing is working fine , if i manually give loadbalancing rule
to Nodeport ip (screenshot attached)
-------
❯ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
AGE
kiran LoadBalancer 10.105.243.145 <pending> 80:30905/TCP
12h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
13h
❯ kubectl describe svc kiran
Name: kiran
Namespace: default
Labels: run=nginx
Annotations: <none>
Selector: run=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.105.243.145
IPs: 10.105.243.145
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30905/TCP
Endpoints: 10.44.0.4:80,10.44.0.5:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From
Message
---- ------ ---- ----
-------
Warning SyncLoadBalancerFailed 35m (x140 over 12h)
service-controller Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: Get
http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA&command=listLoadBalancerRules&keyword=afbeaa6d1594f4e1d9568064fec18b7c&listall=true&response=json&signature=aaY6yZwMlRgzPicp6y4co1rV%2Bl4%3D:
dial tcp 10.102.192.220:8080: i/o timeout
Normal EnsuringLoadBalancer 3m27s (x146 over 12h)
service-controller Ensuring load balancer
-------
On Mon, Aug 1, 2022 at 9:02 AM David Jumani <Da...@shapeblue.com>
wrote:
> Hi Kiran
>
> I've discussed this issue with Pierre offline.
> This happens when the names of the VMs in CloudStack do not match the node
> names in Kubernetes, likely an upper-lower case issue
> A PR to fix it has been raised :
> https://github.com/apache/cloudstack-kubernetes-provider/pull/41
> As a workaround, you can rename the VMs in CloudStack to exactly match the
> output of `kubectl get nodes`
> ________________________________
> From: Kiran manohar Chavala <ki...@gmail.com>
> Sent: Sunday, July 31, 2022 11:21 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Issues with cloudstack-kubernetes-provider load balancer
>
> Hi Pierre Le Fevre
>
> Facing a similar kind of issue when trying to acquire Loadbalncer Ip
>
> Any workaround for the issue
>
> ❯ k describe svc kiran
> Name: kiran
> Namespace: default
> Labels: run=nginx
> Annotations: <none>
> Selector: run=nginx
> Type: LoadBalancer
> IP Family Policy: SingleStack
> IP Families: IPv4
> IP: 10.99.211.32
> IPs: 10.99.211.32
> Port: <unset> 80/TCP
> TargetPort: 8000/TCP
> NodePort: <unset> 31588/TCP
> Endpoints: 10.44.0.4:8000,10.44.0.5:8000
> Session Affinity: None
> External Traffic Policy: Cluster
> Events:
> Type Reason Age From
> Message
> ---- ------ ---- ----
> -------
> Normal EnsuringLoadBalancer 61s (x2 over 2m31s) service-controller
> Ensuring load balancer
> Warning SyncLoadBalancerFailed 31s (x2 over 2m1s) service-controller
> Error syncing load balancer: failed to ensure load balancer: error
> retrieving load balancer rules: Get
>
> http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA&command=listLoadBalancerRules&keyword=a94511f7024b24efdaae576625a6fe68&listall=true&response=json&signature=bpLHBJTA1LgLGV9DCevvKshu2Zg%3D
> :
> dial tcp 10.102.192.220:8080: i/o timeout
>
> Regards
> Kiran
>
>
>
>
>
>
>
>
> On Thu, Jun 16, 2022 at 8:33 PM Pierre Le Fevre <pi...@kth.se> wrote:
>
> > Hi all,
> >
> > I have recently been experimenting with Kubernetes in CloudStack and came
> > across a quite odd problem with the communication
> > between cloudstack-kubernetes-provider and the CloudStack API.
> >
> > After some experimentation with the cloud-config api-url variable, it
> seems
> > that connection should be happening. I tried using wget from within
> > the pods with success.
> > However, whenever services try to sync the load balancer to CloudStack,
> the
> > error* Error syncing load balancer: failed to ensure load balancer: could
> > not find network *shows up.
> >
> > Taking a look at the logs of *cloud-controller-manager*, the same three
> > rows come up repeatedly:
> >
> > I0616 14:02:26.189085 1 event.go:278]
> > Event(v1.ObjectReference{Kind:"Service", Namespace:"default",
> Name:"balls",
> > UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> > ResourceVersion:"1776", FieldPath:""}): type: 'Normal' reason:
> > 'EnsuringLoadBalancer' Ensuring load balancer
> >
> > E0616 14:02:26.273801 1 controller.go:244] error processing service
> > default/balls (will retry): failed to ensure load balancer: could not
> find
> > network
> >
> > I0616 14:02:26.274430 1 event.go:278]
> > Event(v1.ObjectReference{Kind:"Service", Namespace:"default",
> Name:"balls",
> > UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> > ResourceVersion:"1776", FieldPath:""}): type: 'Warning' reason:
> > 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure
> load
> > balancer: could not find network
> >
> >
> > When watching the logs of our management server, the request
> > *listNetworks *appears,
> > coming from the kubernetes IP. It does, however, include an empty *id
> > *parameter ,
> > which in turn returns the error:
> >
> > {
> > "listnetworksresponse": {
> > "uuidList": [],
> > "errorcode": 431,
> > "cserrorcode": 9999,
> > "errortext": "Unable to execute API command listnetworks due to
> > invalid value. Invalid parameter id value= due to incorrect long value
> > format, or entity does not exist or due to incorrect parameter annotation
> > for the field in api cmd class."
> > }
> > }
> >
> > The request: http://
> *domain*/client/api?apiKey=*key*&command=listNetworks&
> > *id=*&response=json&signature=*signature*
> >
> > Looking at the code of cloudstack-kubernetes-provider
> > (cloudstack_loadbalancer.go:417), it seems like the error should return
> > *... **could not find network [network ID]*, but in our case, no ID shows
> > up (it could be nil?).
> > It appears that cs.verifyHosts (cloudstack_loadbalancer.go:100) is
> failing
> > to update lb.networkID, which cascades into this error.
> >
> > Seeing as it shows up even after creating different clusters, could this
> > still be a configuration issue, or is it a known bug in CKS?
> >
> > Thanks in advance,
> > Pierre Le Fevre
> >
>