You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Pierre Le Fevre <pi...@kth.se> on 2022/06/16 15:00:11 UTC
Issues with cloudstack-kubernetes-provider load balancer
Hi all,
I have recently been experimenting with Kubernetes in CloudStack and came
across a quite odd problem with the communication
between cloudstack-kubernetes-provider and the CloudStack API.
After some experimentation with the cloud-config api-url variable, it seems
that connection should be happening. I tried using wget from within
the pods with success.
However, whenever services try to sync the load balancer to CloudStack, the
error* Error syncing load balancer: failed to ensure load balancer: could
not find network *shows up.
Taking a look at the logs of *cloud-controller-manager*, the same three
rows come up repeatedly:
I0616 14:02:26.189085 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
ResourceVersion:"1776", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0616 14:02:26.273801 1 controller.go:244] error processing service
default/balls (will retry): failed to ensure load balancer: could not find
network
I0616 14:02:26.274430 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
ResourceVersion:"1776", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: could not find network
When watching the logs of our management server, the request
*listNetworks *appears,
coming from the kubernetes IP. It does, however, include an empty *id
*parameter ,
which in turn returns the error:
{
"listnetworksresponse": {
"uuidList": [],
"errorcode": 431,
"cserrorcode": 9999,
"errortext": "Unable to execute API command listnetworks due to
invalid value. Invalid parameter id value= due to incorrect long value
format, or entity does not exist or due to incorrect parameter annotation
for the field in api cmd class."
}
}
The request: http://*domain*/client/api?apiKey=*key*&command=listNetworks&
*id=*&response=json&signature=*signature*
Looking at the code of cloudstack-kubernetes-provider
(cloudstack_loadbalancer.go:417), it seems like the error should return
*... **could not find network [network ID]*, but in our case, no ID shows
up (it could be nil?).
It appears that cs.verifyHosts (cloudstack_loadbalancer.go:100) is failing
to update lb.networkID, which cascades into this error.
Seeing as it shows up even after creating different clusters, could this
still be a configuration issue, or is it a known bug in CKS?
Thanks in advance,
Pierre Le Fevre
Re: Issues with cloudstack-kubernetes-provider load balancer
Posted by Kiran manohar Chavala <ki...@gmail.com>.
Hi David
I think the names are fine ,Please find the kubectl output and screenshot
attached
-------
❯ kubectl get nodes
NAME STATUS ROLES AGE
VERSION
kiran-kube-cluster-control-18254bced26 Ready control-plane 13h
v1.24.0
kiran-kube-cluster-node-18254bf3fef Ready <none> 13h
v1.24.0
-------
Also when I give kubectl get svc is the external-ip expected to appear?
The load balancing is working fine , if i manually give loadbalancing rule
to Nodeport ip (screenshot attached)
-------
❯ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
AGE
kiran LoadBalancer 10.105.243.145 <pending> 80:30905/TCP
12h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
13h
❯ kubectl describe svc kiran
Name: kiran
Namespace: default
Labels: run=nginx
Annotations: <none>
Selector: run=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.105.243.145
IPs: 10.105.243.145
Port: <unset> 80/TCP
TargetPort: 80/TCP
NodePort: <unset> 30905/TCP
Endpoints: 10.44.0.4:80,10.44.0.5:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From
Message
---- ------ ---- ----
-------
Warning SyncLoadBalancerFailed 35m (x140 over 12h)
service-controller Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: Get
http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA&command=listLoadBalancerRules&keyword=afbeaa6d1594f4e1d9568064fec18b7c&listall=true&response=json&signature=aaY6yZwMlRgzPicp6y4co1rV%2Bl4%3D:
dial tcp 10.102.192.220:8080: i/o timeout
Normal EnsuringLoadBalancer 3m27s (x146 over 12h)
service-controller Ensuring load balancer
-------
On Mon, Aug 1, 2022 at 9:02 AM David Jumani <Da...@shapeblue.com>
wrote:
> Hi Kiran
>
> I've discussed this issue with Pierre offline.
> This happens when the names of the VMs in CloudStack do not match the node
> names in Kubernetes, likely an upper-lower case issue
> A PR to fix it has been raised :
> https://github.com/apache/cloudstack-kubernetes-provider/pull/41
> As a workaround, you can rename the VMs in CloudStack to exactly match the
> output of `kubectl get nodes`
> ________________________________
> From: Kiran manohar Chavala <ki...@gmail.com>
> Sent: Sunday, July 31, 2022 11:21 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Issues with cloudstack-kubernetes-provider load balancer
>
> Hi Pierre Le Fevre
>
> Facing a similar kind of issue when trying to acquire Loadbalncer Ip
>
> Any workaround for the issue
>
> ❯ k describe svc kiran
> Name: kiran
> Namespace: default
> Labels: run=nginx
> Annotations: <none>
> Selector: run=nginx
> Type: LoadBalancer
> IP Family Policy: SingleStack
> IP Families: IPv4
> IP: 10.99.211.32
> IPs: 10.99.211.32
> Port: <unset> 80/TCP
> TargetPort: 8000/TCP
> NodePort: <unset> 31588/TCP
> Endpoints: 10.44.0.4:8000,10.44.0.5:8000
> Session Affinity: None
> External Traffic Policy: Cluster
> Events:
> Type Reason Age From
> Message
> ---- ------ ---- ----
> -------
> Normal EnsuringLoadBalancer 61s (x2 over 2m31s) service-controller
> Ensuring load balancer
> Warning SyncLoadBalancerFailed 31s (x2 over 2m1s) service-controller
> Error syncing load balancer: failed to ensure load balancer: error
> retrieving load balancer rules: Get
>
> http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA&command=listLoadBalancerRules&keyword=a94511f7024b24efdaae576625a6fe68&listall=true&response=json&signature=bpLHBJTA1LgLGV9DCevvKshu2Zg%3D
> :
> dial tcp 10.102.192.220:8080: i/o timeout
>
> Regards
> Kiran
>
>
>
>
>
>
>
>
> On Thu, Jun 16, 2022 at 8:33 PM Pierre Le Fevre <pi...@kth.se> wrote:
>
> > Hi all,
> >
> > I have recently been experimenting with Kubernetes in CloudStack and came
> > across a quite odd problem with the communication
> > between cloudstack-kubernetes-provider and the CloudStack API.
> >
> > After some experimentation with the cloud-config api-url variable, it
> seems
> > that connection should be happening. I tried using wget from within
> > the pods with success.
> > However, whenever services try to sync the load balancer to CloudStack,
> the
> > error* Error syncing load balancer: failed to ensure load balancer: could
> > not find network *shows up.
> >
> > Taking a look at the logs of *cloud-controller-manager*, the same three
> > rows come up repeatedly:
> >
> > I0616 14:02:26.189085 1 event.go:278]
> > Event(v1.ObjectReference{Kind:"Service", Namespace:"default",
> Name:"balls",
> > UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> > ResourceVersion:"1776", FieldPath:""}): type: 'Normal' reason:
> > 'EnsuringLoadBalancer' Ensuring load balancer
> >
> > E0616 14:02:26.273801 1 controller.go:244] error processing service
> > default/balls (will retry): failed to ensure load balancer: could not
> find
> > network
> >
> > I0616 14:02:26.274430 1 event.go:278]
> > Event(v1.ObjectReference{Kind:"Service", Namespace:"default",
> Name:"balls",
> > UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> > ResourceVersion:"1776", FieldPath:""}): type: 'Warning' reason:
> > 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure
> load
> > balancer: could not find network
> >
> >
> > When watching the logs of our management server, the request
> > *listNetworks *appears,
> > coming from the kubernetes IP. It does, however, include an empty *id
> > *parameter ,
> > which in turn returns the error:
> >
> > {
> > "listnetworksresponse": {
> > "uuidList": [],
> > "errorcode": 431,
> > "cserrorcode": 9999,
> > "errortext": "Unable to execute API command listnetworks due to
> > invalid value. Invalid parameter id value= due to incorrect long value
> > format, or entity does not exist or due to incorrect parameter annotation
> > for the field in api cmd class."
> > }
> > }
> >
> > The request: http://
> *domain*/client/api?apiKey=*key*&command=listNetworks&
> > *id=*&response=json&signature=*signature*
> >
> > Looking at the code of cloudstack-kubernetes-provider
> > (cloudstack_loadbalancer.go:417), it seems like the error should return
> > *... **could not find network [network ID]*, but in our case, no ID shows
> > up (it could be nil?).
> > It appears that cs.verifyHosts (cloudstack_loadbalancer.go:100) is
> failing
> > to update lb.networkID, which cascades into this error.
> >
> > Seeing as it shows up even after creating different clusters, could this
> > still be a configuration issue, or is it a known bug in CKS?
> >
> > Thanks in advance,
> > Pierre Le Fevre
> >
>
Re: Issues with cloudstack-kubernetes-provider load balancer
Posted by David Jumani <Da...@shapeblue.com>.
Hi Kiran
I've discussed this issue with Pierre offline.
This happens when the names of the VMs in CloudStack do not match the node names in Kubernetes, likely an upper-lower case issue
A PR to fix it has been raised : https://github.com/apache/cloudstack-kubernetes-provider/pull/41
As a workaround, you can rename the VMs in CloudStack to exactly match the output of `kubectl get nodes`
________________________________
From: Kiran manohar Chavala <ki...@gmail.com>
Sent: Sunday, July 31, 2022 11:21 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Issues with cloudstack-kubernetes-provider load balancer
Hi Pierre Le Fevre
Facing a similar kind of issue when trying to acquire Loadbalncer Ip
Any workaround for the issue
❯ k describe svc kiran
Name: kiran
Namespace: default
Labels: run=nginx
Annotations: <none>
Selector: run=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.99.211.32
IPs: 10.99.211.32
Port: <unset> 80/TCP
TargetPort: 8000/TCP
NodePort: <unset> 31588/TCP
Endpoints: 10.44.0.4:8000,10.44.0.5:8000
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From
Message
---- ------ ---- ----
-------
Normal EnsuringLoadBalancer 61s (x2 over 2m31s) service-controller
Ensuring load balancer
Warning SyncLoadBalancerFailed 31s (x2 over 2m1s) service-controller
Error syncing load balancer: failed to ensure load balancer: error
retrieving load balancer rules: Get
http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA&command=listLoadBalancerRules&keyword=a94511f7024b24efdaae576625a6fe68&listall=true&response=json&signature=bpLHBJTA1LgLGV9DCevvKshu2Zg%3D:
dial tcp 10.102.192.220:8080: i/o timeout
Regards
Kiran
On Thu, Jun 16, 2022 at 8:33 PM Pierre Le Fevre <pi...@kth.se> wrote:
> Hi all,
>
> I have recently been experimenting with Kubernetes in CloudStack and came
> across a quite odd problem with the communication
> between cloudstack-kubernetes-provider and the CloudStack API.
>
> After some experimentation with the cloud-config api-url variable, it seems
> that connection should be happening. I tried using wget from within
> the pods with success.
> However, whenever services try to sync the load balancer to CloudStack, the
> error* Error syncing load balancer: failed to ensure load balancer: could
> not find network *shows up.
>
> Taking a look at the logs of *cloud-controller-manager*, the same three
> rows come up repeatedly:
>
> I0616 14:02:26.189085 1 event.go:278]
> Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
> UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> ResourceVersion:"1776", FieldPath:""}): type: 'Normal' reason:
> 'EnsuringLoadBalancer' Ensuring load balancer
>
> E0616 14:02:26.273801 1 controller.go:244] error processing service
> default/balls (will retry): failed to ensure load balancer: could not find
> network
>
> I0616 14:02:26.274430 1 event.go:278]
> Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
> UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> ResourceVersion:"1776", FieldPath:""}): type: 'Warning' reason:
> 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
> balancer: could not find network
>
>
> When watching the logs of our management server, the request
> *listNetworks *appears,
> coming from the kubernetes IP. It does, however, include an empty *id
> *parameter ,
> which in turn returns the error:
>
> {
> "listnetworksresponse": {
> "uuidList": [],
> "errorcode": 431,
> "cserrorcode": 9999,
> "errortext": "Unable to execute API command listnetworks due to
> invalid value. Invalid parameter id value= due to incorrect long value
> format, or entity does not exist or due to incorrect parameter annotation
> for the field in api cmd class."
> }
> }
>
> The request: http://*domain*/client/api?apiKey=*key*&command=listNetworks&
> *id=*&response=json&signature=*signature*
>
> Looking at the code of cloudstack-kubernetes-provider
> (cloudstack_loadbalancer.go:417), it seems like the error should return
> *... **could not find network [network ID]*, but in our case, no ID shows
> up (it could be nil?).
> It appears that cs.verifyHosts (cloudstack_loadbalancer.go:100) is failing
> to update lb.networkID, which cascades into this error.
>
> Seeing as it shows up even after creating different clusters, could this
> still be a configuration issue, or is it a known bug in CKS?
>
> Thanks in advance,
> Pierre Le Fevre
>
Re: Issues with cloudstack-kubernetes-provider load balancer
Posted by Kiran manohar Chavala <ki...@gmail.com>.
Hi Pierre Le Fevre
Facing a similar kind of issue when trying to acquire Loadbalncer Ip
Any workaround for the issue
❯ k describe svc kiran
Name: kiran
Namespace: default
Labels: run=nginx
Annotations: <none>
Selector: run=nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.99.211.32
IPs: 10.99.211.32
Port: <unset> 80/TCP
TargetPort: 8000/TCP
NodePort: <unset> 31588/TCP
Endpoints: 10.44.0.4:8000,10.44.0.5:8000
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From
Message
---- ------ ---- ----
-------
Normal EnsuringLoadBalancer 61s (x2 over 2m31s) service-controller
Ensuring load balancer
Warning SyncLoadBalancerFailed 31s (x2 over 2m1s) service-controller
Error syncing load balancer: failed to ensure load balancer: error
retrieving load balancer rules: Get
http://10.102.192.220:8080/client/api?apiKey=ih7O6aKn0-LD7FL2Y6H3wfXv0B1iRXQOG7rZc5iip__eNmiRpKx_27dF5sy_6kJ-7OZt7mx7w9wZHu9Pq9bRfA&command=listLoadBalancerRules&keyword=a94511f7024b24efdaae576625a6fe68&listall=true&response=json&signature=bpLHBJTA1LgLGV9DCevvKshu2Zg%3D:
dial tcp 10.102.192.220:8080: i/o timeout
Regards
Kiran
On Thu, Jun 16, 2022 at 8:33 PM Pierre Le Fevre <pi...@kth.se> wrote:
> Hi all,
>
> I have recently been experimenting with Kubernetes in CloudStack and came
> across a quite odd problem with the communication
> between cloudstack-kubernetes-provider and the CloudStack API.
>
> After some experimentation with the cloud-config api-url variable, it seems
> that connection should be happening. I tried using wget from within
> the pods with success.
> However, whenever services try to sync the load balancer to CloudStack, the
> error* Error syncing load balancer: failed to ensure load balancer: could
> not find network *shows up.
>
> Taking a look at the logs of *cloud-controller-manager*, the same three
> rows come up repeatedly:
>
> I0616 14:02:26.189085 1 event.go:278]
> Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
> UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> ResourceVersion:"1776", FieldPath:""}): type: 'Normal' reason:
> 'EnsuringLoadBalancer' Ensuring load balancer
>
> E0616 14:02:26.273801 1 controller.go:244] error processing service
> default/balls (will retry): failed to ensure load balancer: could not find
> network
>
> I0616 14:02:26.274430 1 event.go:278]
> Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"balls",
> UID:"f75f1f54-2c97-4417-946d-90a536087ad8", APIVersion:"v1",
> ResourceVersion:"1776", FieldPath:""}): type: 'Warning' reason:
> 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
> balancer: could not find network
>
>
> When watching the logs of our management server, the request
> *listNetworks *appears,
> coming from the kubernetes IP. It does, however, include an empty *id
> *parameter ,
> which in turn returns the error:
>
> {
> "listnetworksresponse": {
> "uuidList": [],
> "errorcode": 431,
> "cserrorcode": 9999,
> "errortext": "Unable to execute API command listnetworks due to
> invalid value. Invalid parameter id value= due to incorrect long value
> format, or entity does not exist or due to incorrect parameter annotation
> for the field in api cmd class."
> }
> }
>
> The request: http://*domain*/client/api?apiKey=*key*&command=listNetworks&
> *id=*&response=json&signature=*signature*
>
> Looking at the code of cloudstack-kubernetes-provider
> (cloudstack_loadbalancer.go:417), it seems like the error should return
> *... **could not find network [network ID]*, but in our case, no ID shows
> up (it could be nil?).
> It appears that cs.verifyHosts (cloudstack_loadbalancer.go:100) is failing
> to update lb.networkID, which cascades into this error.
>
> Seeing as it shows up even after creating different clusters, could this
> still be a configuration issue, or is it a known bug in CKS?
>
> Thanks in advance,
> Pierre Le Fevre
>