You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/12/30 02:25:42 UTC
[GitHub] [apisix-helm-chart] jishaashokan opened a new issue, #431: request help: apisix-etcd-1 is unhealthy: failed to commit proposal: context deadline exceeded
jishaashokan opened a new issue, #431:
URL: https://github.com/apache/apisix-helm-chart/issues/431
### Issue description
apisix was installed using :
git clone https://github.com/apache/apisix-helm-chart.git
helm repo add bitnami https://charts.bitnami.com/bitnami
helm dependency update ./charts/apisix
helm install apisix apisix/apisix \
--set gateway.type=LoadBalancer \
--set allow.ipList="{0.0.0.0/0}" \
--set ingress-controller.enabled=true \
--namespace ingress-apisix \
--set ingress-controller.config.apisix.serviceNamespace=ingress-apisix \
--set gateway.tls.enabled=true \
--set ingress-controller.config.apisix.adminKey=xxxx \
--set admin.credentials.admin=xxxx \
--set admin.credentials.viewer=yyyy \
--set ingressController.config.apisix.baseURL=http://apisix-admin:9180/apisix/admin
The following pods come up, but apisix-etcd-1 crashes from the beginning.
NAME READY STATUS RESTARTS AGE
apisix-54cb677c56-n8jjp 1/1 Running 0 178m
apisix-dashboard-78dc4dc8bb-h9bnb 1/1 Running 2 (106m ago) 2d
apisix-etcd-0 1/1 Running 0 161m
apisix-etcd-1 0/1 CrashLoopBackOff 21 (3m45s ago) 115m
apisix-etcd-2 1/1 Running 0 70m
kubectl exec -it apisix-etcd-2 -- etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 4.01932ms
kubectl exec -it apisix-etcd-1 -- etcdctl endpoint health
{"level":"warn","ts":"2022-12-28T07:49:45.052Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002e6a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
command terminated with exit code 1
kubectl exec -it apisix-etcd-0 -- etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.962883ms
Also, after installing Apisix Route, Upstream and TLS, the apisix-ingress-controller logs gives the following errors :
kubectl logs -f apisix-ingress-controller-6c4fbd9697-q9whw
2022-12-28T13:30:17+08:00 error apisix/upstream.go:75 failed to get upstream from APISIX {"name": "ingress-apisix_apisix-etcd_2379", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/29b61b45", "cluster": "default", "error": "Get \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/29b61b45\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2022-12-28T13:30:17+08:00 error types/types.go:218 failed to get upstream {"upstream": "ingress-apisix_apisix-etcd_2379", "cluster": "name=default; base_url=http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin", "error": "Get \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/29b61b45\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2022-12-28T13:30:17+08:00 warn endpoint/endpoint.go:130 sync endpoints failed, will retry {"object": {"Type":2,"Object":{},"OldObject":null,"Tombstone":null}}
2022-12-28T13:30:17+08:00 error apisix/route.go:163 failed to create route: Put "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/9a4e702d": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-12-28T13:30:17+08:00 warn apisix/apisix_route.go:513 sync ApisixRoute failed, will retry {"object": {"Type":1,"Object":{"Key":"ingress-apisix/grpc-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "2 errors occurred:\n\t* Put \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/69084189\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n\t* Put \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/9a4e702d\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n\n"}
2022-12-28T13:30:18+08:00 error apisix/apisix_tls.go:213 failed to sync SSL to APISIX {"error": "Put \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/ssls/46a9cf0b\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "ssl": {"id":"46a9cf0b","snis":["grpc-proxy"],"cert":"
### Environment
- your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
- your Kubernetes cluster version (output of kubectl version):
- if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tao12345666333 commented on issue #431: request help: apisix-etcd-1 is unhealthy: failed to commit proposal: context deadline exceeded
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #431:
URL: https://github.com/apache/apisix-helm-chart/issues/431#issuecomment-1367689423
```
apisix-etcd-1 0/1 CrashLoopBackOff 21 (3m45s ago) 115m
```
you can check the pod's log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org