You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/05/08 04:19:08 UTC
[GitHub] [apisix-ingress-controller] pioneer-hash opened a new issue #427: request help: failed to wait the default cluster to be ready:
pioneer-hash opened a new issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427
### Issue description
A few days ago, when the deployment was completed, there was no problem in synchronizing the data. Now, there was an error when synchronizing the configuration. Could you please help me to see what caused the error
```
E0508 11:58:49.908189 1 leaderelection.go:325] error retrieving resource lock gray-publish/ingress-apisix-leader: Get https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/gray-publish/leases/ingress-apisix-leader: context deadline exceeded
2021-05-08T11:58:49+08:00 info controller/controller.go:202 LeaderElection {"message": "sitech-gray-ingress-controller-7779b6bd88-chpnm stopped leading", "event_type": "Normal"}
I0508 11:58:49.908333 1 leaderelection.go:278] failed to renew lease gray-publish/ingress-apisix-leader: timed out waiting for the condition
E0508 11:58:49.908405 1 leaderelection.go:301] Failed to release lock: resource name may not be empty
2021-05-08T11:58:49+08:00 info controller/controller.go:249 controller now is running as a candidate {"namespace": "gray-publish", "pod": "sitech-gray-ingress-controller-7779b6bd88-chpnm"}
I0508 11:58:49.908450 1 leaderelection.go:243] attempting to acquire leader lease gray-publish/ingress-apisix-leader...
[GIN] 2021/05/08 - 11:58:50 | 200 | 64.605µs | 172.21.9.40 | GET "/healthz"
2021-05-08T11:58:52+08:00 info controller/controller.go:202 LeaderElection {"message": "sitech-gray-ingress-controller-7779b6bd88-chpnm became leader", "event_type": "Normal"}
I0508 11:58:52.561077 1 leaderelection.go:253] successfully acquired lease gray-publish/ingress-apisix-leader
2021-05-08T11:58:52+08:00 info controller/controller.go:277 controller now is running as leader {"namespace": "gray-publish", "pod": "sitech-gray-ingress-controller-7779b6bd88-chpnm"}
2021-05-08T11:58:52+08:00 error controller/controller.go:296 failed to wait the default cluster to be ready: Get http://sitech-gray-admin:9180/apisix/admin/routes: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
```
I accessed apisix-ingress-controller and it was successful
```
/ingress-apisix # wget http://apisix-admin:9180/apisix/admin/routes --header="X-API-Key: edd1c9f034335f136f87ad84b625c8f1"
wget: bad address 'apisix-admin:9180'
/ingress-apisix # wget http://sitech-gray-admin:9180/apisix/admin/routes --header="X-API-Key: edd1c9f034335f136f87ad84b625c8f1"
Connecting to sitech-gray-admin:9180 (10.111.0.24:9180)
saving to 'routes'
routes 100% |******************************************************************************************************| 908 0:00:00 ETA
'routes' saved
/ingress-apisix # ls
apisix-ingress-controller conf routes
```
### Environment
* your apisix-ingress-controller version (output of `apisix-ingress-controller version --long`);
*apisix-ingress-controller version :0.5.0
apisx-version:2.5
* your Kubernetes cluster version (output of `kubectl version`);
* if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (`uname -a`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835151661
OK, the apisix base url `sitech-gray-admin` confuses me.
From the error log, your controller cannot communicate to apisix admin, could you try to send request to `sitech-gray-admin` from the controller container?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] pioneer-hash commented on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
pioneer-hash commented on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835128769
> This line:
>
> ```
> E0508 11:58:49.908189 1 leaderelection.go:325] error retrieving resource lock gray-publish/ingress-apisix-leader: Get https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/gray-publish/leases/ingress-apisix-leader: context deadline exceeded
> ```
>
> is the root cause, it seems your kube-apiserver is not running normally.
Could you please tell me what command I can use to query the status of kube-apiserver
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] pioneer-hash commented on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
pioneer-hash commented on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835155543
> OK, the apisix base url `sitech-gray-admin` confuses me.
>
> From the error log, your controller cannot communicate to apisix admin, could you try to send request to `sitech-gray-admin` from the controller container?
I I go inside pod of apisix-ingress-controller and get Route information directly it was successful
```
/ingress-apisix # wget http://apisix-admin:9180/apisix/admin/routes --header="X-API-Key: edd1c9f034335f136f87ad84b625c8f1"
wget: bad address 'apisix-admin:9180'
/ingress-apisix # wget http://sitech-gray-admin:9180/apisix/admin/routes --header="X-API-Key: edd1c9f034335f136f87ad84b625c8f1"
Connecting to sitech-gray-admin:9180 (10.111.0.24:9180)
saving to 'routes'
routes 100% |******************************************************************************************************| 908 0:00:00 ETA
'routes' saved
/ingress-apisix # ls
apisix-ingress-controller conf routes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] pioneer-hash edited a comment on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
pioneer-hash edited a comment on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835128769
> This line:
>
> ```
> E0508 11:58:49.908189 1 leaderelection.go:325] error retrieving resource lock gray-publish/ingress-apisix-leader: Get https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/gray-publish/leases/ingress-apisix-leader: context deadline exceeded
> ```
>
> is the root cause, it seems your kube-apiserver is not running normally.
I ues kubectl get pods -n kube-system not found abnormal
```
[root@host-172-21-9-35 ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-688c5dc8c7-mlntg 1/1 Running 8 103d
calico-node-2888c 1/1 Running 0 101d
calico-node-g74r9 1/1 Running 0 16d
calico-node-gd6tc 1/1 Running 7 101d
calico-node-zl8pv 1/1 Running 0 101d
coredns-6955765f44-jzndk 1/1 Running 7 103d
coredns-6955765f44-p8t8j 1/1 Running 6 103d
etcd-host-172-21-9-35 1/1 Running 8 101d
etcd-host-172-21-9-36 1/1 Running 6 101d
etcd-host-172-21-9-37 1/1 Running 5 101d
kube-apiserver-host-172-21-9-35 1/1 Running 11 103d
kube-apiserver-host-172-21-9-36 1/1 Running 6 103d
kube-apiserver-host-172-21-9-37 1/1 Running 4 103d
kube-controller-manager-host-172-21-9-35 1/1 Running 222 101d
kube-controller-manager-host-172-21-9-36 1/1 Running 216 101d
kube-controller-manager-host-172-21-9-37 1/1 Running 213 101d
kube-proxy-jt6wz 1/1 Running 0 16d
kube-proxy-l6ccf 1/1 Running 0 101d
kube-proxy-r8p7z 1/1 Running 8 101d
kube-proxy-rx8vs 1/1 Running 0 101d
kube-scheduler-host-172-21-9-35 1/1 Running 223 103d
kube-scheduler-host-172-21-9-36 1/1 Running 214 103d
kube-scheduler-host-172-21-9-37 1/1 Running 221 103d
kube-sealyun-lvscare-host-172-21-9-40 1/1 Running 0 16d
```
Could you please tell me what command I can use to query the status of kube-apiserver
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] pioneer-hash commented on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
pioneer-hash commented on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835149725
> How did you configure apisix-ingress-controller? Especially the baseURL.
apisix install
```
helm install sitech-gray . -n gray-publish \
--set image.repository="172.21.9.35:30002/gray-stare/gray" \
--set image.tag="2.5-centos" \
--set etcd.enabled=false \
--set etcd.host={http://172.21.9.36:22379} \
--set etcd.user="root" \
--set etcd.password="123" \
--set etcd.prefix="/apisix" \
--set allow.ipList=""
```
apisix-ingress-controller install
```
helm install sitech-gray-ingress-controller ./apisix-ingress-controller \
--set config.apisix.baseURL=http://sitech-gray-admin:9180/apisix/admin \
--set config.apisix.adminKey=edd1c9f034335f136f87ad84b625c8f1 \
--set image.repository="172.21.9.35:30002/gray-stare/gray-ingress" \
--set image.tag="0.5.0" \
--namespace gray-publish
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-1075909523
You can try deploying with the new version and see if the problem persists.
I'll close this issue because it's been too long. If your issue is not resolved, feel free to reopen it, or create a new issue.
Thanks for your report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 closed issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
tao12345666333 closed issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] pioneer-hash edited a comment on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
pioneer-hash edited a comment on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835155543
> OK, the apisix base url `sitech-gray-admin` confuses me.
>
> From the error log, your controller cannot communicate to apisix admin, could you try to send request to `sitech-gray-admin` from the controller container?
I I go inside pod of apisix-ingress-controller and get Route information directly it was successful
```
/ingress-apisix # wget http://sitech-gray-admin:9180/apisix/admin/routes --header="X-API-Key: edd1c9f034335f136f87ad84b625c8f1"
Connecting to sitech-gray-admin:9180 (10.111.0.24:9180)
saving to 'routes'
routes 100% |******************************************************************************************************| 908 0:00:00 ETA
'routes' saved
/ingress-apisix # ls
apisix-ingress-controller conf routes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835147698
> 2021-05-08T11:58:52+08:00 error controller/controller.go:296 failed to wait the default cluster to be ready: Get http://sitech-gray-admin:9180/apisix/admin/routes: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
How did you configure apisix-ingress-controller? Especially the baseURL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835107161
This line:
```
E0508 11:58:49.908189 1 leaderelection.go:325] error retrieving resource lock gray-publish/ingress-apisix-leader: Get https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/gray-publish/leases/ingress-apisix-leader: context deadline exceeded
```
is the root cause, it seems your kube-apiserver is not running normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] pioneer-hash edited a comment on issue #427: request help: failed to wait the default cluster to be ready:
Posted by GitBox <gi...@apache.org>.
pioneer-hash edited a comment on issue #427:
URL: https://github.com/apache/apisix-ingress-controller/issues/427#issuecomment-835155543
> OK, the apisix base url `sitech-gray-admin` confuses me.
>
> From the error log, your controller cannot communicate to apisix admin, could you try to send request to `sitech-gray-admin` from the controller container?
I I go inside pod of apisix-ingress-controller and get Route information directly it was successful
```
/ingress-apisix # wget http://sitech-gray-admin:9180/apisix/admin/routes --header="X-API-Key: edd1c9f034335f136f87ad84b625c8f1"
Connecting to sitech-gray-admin:9180 (10.111.0.24:9180)
saving to 'routes'
routes 100% |******************************************************************************************************| 908 0:00:00 ETA
'routes' saved
/ingress-apisix # ls
apisix-ingress-controller conf routes
```
Let me just add one more thing : This deployment scenario has previously been able to successfully synchronize configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org