You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/12/28 08:03:47 UTC
[GitHub] [apisix-ingress-controller] jishaashokan opened a new issue, #1565: request help: apisix-etcd-1 is unhealthy: failed to commit proposal: context deadline exceeded
jishaashokan opened a new issue, #1565:
URL: https://github.com/apache/apisix-ingress-controller/issues/1565
### Issue description
apisix was installed using :
git clone https://github.com/apache/apisix-helm-chart.git
helm repo add bitnami https://charts.bitnami.com/bitnami
helm dependency update ./charts/apisix
helm install apisix apisix/apisix \
--set gateway.type=LoadBalancer \
--set allow.ipList="{0.0.0.0/0}" \
--set ingress-controller.enabled=true \
--namespace ingress-apisix \
--set ingress-controller.config.apisix.serviceNamespace=ingress-apisix \
--set gateway.tls.enabled=true \
--set ingress-controller.config.apisix.adminKey=xxxx \
--set admin.credentials.admin=xxxx \
--set admin.credentials.viewer=yyyy \
--set ingressController.config.apisix.baseURL=http://apisix-admin:9180/apisix/admin
The following pods come up, but apisix-etcd-1 crashes from the beginning.
NAME READY STATUS RESTARTS AGE
apisix-54cb677c56-n8jjp 1/1 Running 0 178m
apisix-dashboard-78dc4dc8bb-h9bnb 1/1 Running 2 (106m ago) 2d
apisix-etcd-0 1/1 Running 0 161m
apisix-etcd-1 0/1 CrashLoopBackOff 21 (3m45s ago) 115m
apisix-etcd-2 1/1 Running 0 70m
kubectl exec -it apisix-etcd-2 -- etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 4.01932ms
kubectl exec -it apisix-etcd-1 -- etcdctl endpoint health
{"level":"warn","ts":"2022-12-28T07:49:45.052Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002e6a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
command terminated with exit code 1
kubectl exec -it apisix-etcd-0 -- etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.962883ms
Also, after installing Apisix Route, Upstream and TLS, the apisix-ingress-controller logs gives the following errors :
kubectl logs -f apisix-ingress-controller-6c4fbd9697-q9whw
2022-12-28T13:30:17+08:00 error apisix/upstream.go:75 failed to get upstream from APISIX {"name": "ingress-apisix_apisix-etcd_2379", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/29b61b45", "cluster": "default", "error": "Get \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/29b61b45\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2022-12-28T13:30:17+08:00 error types/types.go:218 failed to get upstream {"upstream": "ingress-apisix_apisix-etcd_2379", "cluster": "name=default; base_url=http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin", "error": "Get \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/29b61b45\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2022-12-28T13:30:17+08:00 warn endpoint/endpoint.go:130 sync endpoints failed, will retry {"object": {"Type":2,"Object":{},"OldObject":null,"Tombstone":null}}
2022-12-28T13:30:17+08:00 error apisix/route.go:163 failed to create route: Put "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/9a4e702d": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-12-28T13:30:17+08:00 warn apisix/apisix_route.go:513 sync ApisixRoute failed, will retry {"object": {"Type":1,"Object":{"Key":"ingress-apisix/grpc-route","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "2 errors occurred:\n\t* Put \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/69084189\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n\t* Put \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/9a4e702d\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n\n"}
2022-12-28T13:30:18+08:00 error apisix/apisix_tls.go:213 failed to sync SSL to APISIX {"error": "Put \"http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/ssls/46a9cf0b\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "ssl": {"id":"46a9cf0b","snis":["grpc-proxy"],"cert":"
### Environment
- your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
- your Kubernetes cluster version (output of kubectl version):
- if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] jishaashokan commented on issue #1565: request help: apisix-etcd-1 is unhealthy: failed to commit proposal: context deadline exceeded
Posted by GitBox <gi...@apache.org>.
jishaashokan commented on issue #1565:
URL: https://github.com/apache/apisix-ingress-controller/issues/1565#issuecomment-1366495054
This example fails : https://apisix.apache.org/docs/ingress-controller/tutorials/proxy-grpc-service/
`kubectl run -it -n ingress-apisix --rm grpcurl --restart=Never --image=quay.io/mhausenblas/gump:0.1 -- sh
If you don't see a command prompt, try pressing enter.
/go $ grpcurl --plaintext yages:9000 yages.Echo.Ping
{
"text": "pong"
}
/go $ grpcurl --insecure -servername grpc-proxy apisix-gateway:443 yages.Echo.Ping
Error invoking method "yages.Echo.Ping": failed to query for service descriptor "yages.Echo": rpc error: code = Internal desc = transport: received the unexpected content-type "text/plain; charset=utf-8"
/go $ exit
`
`mk describe au yages
Name: yages
Namespace: ingress-apisix
Labels: <none>
Annotations: <none>
API Version: apisix.apache.org/v2
Kind: ApisixUpstream
Metadata:
Creation Timestamp: 2022-12-28T08:14:18Z
Generation: 1
Managed Fields:
API Version: apisix.apache.org/v2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:scheme:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-12-28T08:14:18Z
Resource Version: 72572830
UID: e172ed55-f3a7-4dcd-af6d-96c1a93d24ea
Spec:
Scheme: grpc
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ResourcesSynced 2m31s (x6 over 24m) ApisixIngress ApisixIngress synced successfully
`
`mk describe ar grpc-proxy-route
Name: grpc-proxy-route
Namespace: ingress-apisix
Labels: <none>
Annotations: <none>
API Version: apisix.apache.org/v2
Kind: ApisixRoute
Metadata:
Creation Timestamp: 2022-12-28T08:13:49Z
Generation: 1
Managed Fields:
API Version: apisix.apache.org/v2
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
Manager: apisix-ingress-controller
Operation: Update
Subresource: status
Time: 2022-12-28T08:13:49Z
API Version: apisix.apache.org/v2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:http:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-12-28T08:13:49Z
Resource Version: 72572691
UID: 391fe005-7ca8-4a5a-96c2-f493c796a5d6
Spec:
Http:
Backends:
Service Name: yages
Service Port: 9000
Weight: 10
Match:
Hosts:
grpc-proxy
Paths:
/*
Name: grpc-route
Status:
Conditions:
Message: Sync Successfully
Observed Generation: 1
Reason: ResourcesSynced
Status: True
Type: ResourcesAvailable
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ResourcesSynced 69s (x7 over 23m) ApisixIngress ApisixIngress synced successfully
`
`mk describe apisixtls.apisix.apache.org/grpc-secret
Name: grpc-secret
Namespace: ingress-apisix
Labels: <none>
Annotations: <none>
API Version: apisix.apache.org/v2
Kind: ApisixTls
Metadata:
Creation Timestamp: 2022-12-28T08:14:49Z
Generation: 1
Managed Fields:
API Version: apisix.apache.org/v2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:hosts:
f:secret:
.:
f:name:
f:namespace:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-12-28T08:14:49Z
API Version: apisix.apache.org/v2
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
Manager: apisix-ingress-controller
Operation: Update
Subresource: status
Time: 2022-12-28T08:14:52Z
Resource Version: 72573006
UID: 32f8e955-ea77-4d58-8179-2529cf9be450
Spec:
Hosts:
grpc-proxy
Secret:
Name: grpc-secret
Namespace: ingress-apisix
Status:
Conditions:
Last Transition Time: 2022-12-28T08:14:52Z
Message: Sync Successfully
Observed Generation: 1
Reason: ResourcesSynced
Status: True
Type: ResourcesAvailable
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ResourcesSynced 4m32s (x14 over 61m) ApisixIngress ApisixIngress synced successfully
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] jishaashokan commented on issue #1565: request help: apisix-etcd-1 is unhealthy: failed to commit proposal: context deadline exceeded
Posted by GitBox <gi...@apache.org>.
jishaashokan commented on issue #1565:
URL: https://github.com/apache/apisix-ingress-controller/issues/1565#issuecomment-1366474415
/ingress-apisix # ./apisix-ingress-controller version --long
Version: 1.6.0
Git SHA: no-git-module
Go Version: go1.19.4
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64
Kubernetes version
1.22
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org