You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/04/22 16:03:35 UTC
[GitHub] [apisix-helm-chart] JohnnyQianP opened a new issue, #272: Request help: apisix-etcd Failed to bind volumes: timed out waiting for the condition
JohnnyQianP opened a new issue, #272:
URL: https://github.com/apache/apisix-helm-chart/issues/272
```
# kubectl get pods -n ingress-apisix
NAME READY STATUS RESTARTS AGE
my-apisix-5fbc74568f-6pps2 0/1 Init:0/1 0 26m
my-apisix-etcd-0 0/1 Pending 0 50m
my-apisix-etcd-1 0/1 Pending 0 50m
my-apisix-etcd-2 0/1 Pending 0 25m
# kubectl describe pod my-apisix-etcd-0 -n ingress-apisix
Name: my-apisix-etcd-0
Namespace: ingress-apisix
Priority: 0
Node: <none>
Labels: app.kubernetes.io/instance=my-apisix
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=etcd
controller-revision-hash=my-apisix-etcd-5b96d6bfb4
helm.sh/chart=etcd-7.0.4
statefulset.kubernetes.io/pod-name=my-apisix-etcd-0
Annotations: checksum/token-secret: 971aed8b8aa5756f775f81a27b2e5d07dae029e5ed787033e6703db5be62b008
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/my-apisix-etcd
Containers:
etcd:
Image: docker.io/bitnami/etcd:3.4.18-debian-10-r14
Ports: 2379/TCP, 2380/TCP
Host Ports: 0/TCP, 0/TCP
Liveness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: my-apisix-etcd-0 (v1:metadata.name)
MY_STS_NAME: my-apisix-etcd
ETCDCTL_API: 3
ETCD_ON_K8S: yes
ETCD_START_FROM_SNAPSHOT: no
ETCD_DISASTER_RECOVERY: no
ETCD_NAME: $(MY_POD_NAME)
ETCD_DATA_DIR: /bitnami/etcd/data
ETCD_LOG_LEVEL: info
ALLOW_NONE_AUTHENTICATION: yes
ETCD_AUTH_TOKEN: jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
ETCD_ADVERTISE_CLIENT_URLS: http://$(MY_POD_NAME).my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://my-apisix-etcd.ingress-apisix.svc.cluster.local:2379
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://$(MY_POD_NAME).my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER: my-apisix-etcd-0=http://my-apisix-etcd-0.my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,my-apisix-etcd-1=http://my-apisix-etcd-1.my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,my-apisix-etcd-2=http://my-apisix-etcd-2.my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_CLUSTER_DOMAIN: my-apisix-etcd-headless.ingress-apisix.svc.cluster.local
Mounts:
/bitnami/etcd from data (rw)
/opt/bitnami/etcd/certs/token/ from etcd-jwt-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-2756j (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-my-apisix-etcd-0
ReadOnly: false
etcd-jwt-token:
Type: Secret (a volume populated by a Secret)
SecretName: my-apisix-etcd-jwt-token
Optional: false
default-token-2756j:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-2756j
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler Failed to bind volumes: timed out waiting for the condition
Warning FailedScheduling <unknown> default-scheduler Failed to bind volumes: timed out waiting for the condition
Warning FailedScheduling <unknown> default-scheduler Failed to bind volumes: timed out waiting for the condition
Warning FailedScheduling <unknown> default-scheduler Failed to bind volumes: timed out waiting for the condition
```
Any ideas? Please help to find the solution. I guess it's related with etcd, but I don't know the details.
Below, apisix is installed
# helm3 list -n ingress-apisix
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
my-apisix ingress-apisix 1 2022-04-22 22:48:08.038264423 +0800 CST deployed apisix-0.9.1 2.13.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] JohnnyQianP commented on issue #272: Request help: apisix-etcd Failed to bind volumes: timed out waiting for the condition
Posted by GitBox <gi...@apache.org>.
JohnnyQianP commented on issue #272:
URL: https://github.com/apache/apisix-helm-chart/issues/272#issuecomment-1107349261
@tao12345666333
I check it below:
```
kubectl get sts -n default
NAME READY AGE
apisix-etcd 1/1 18h
```
statefulset exist, but container pods throw exception
And the service also has an exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tao12345666333 commented on issue #272: Request help: apisix-etcd Failed to bind volumes: timed out waiting for the condition
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #272:
URL: https://github.com/apache/apisix-helm-chart/issues/272#issuecomment-1107353596
There is already a clear message
Failed to bind volumes
in the error log.
You should check your environments, PV,PVC,SC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tao12345666333 commented on issue #272: Request help: apisix-etcd Failed to bind volumes: timed out waiting for the condition
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #272:
URL: https://github.com/apache/apisix-helm-chart/issues/272#issuecomment-1107440997
This means that there are some network and configuration related issues in your cluster.
plz check the document for details
https://kubernetes.io/docs/concepts/services-networking/service/#headless-services
This has nothing to do with APISIX or bitnami/etcd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] JohnnyQianP commented on issue #272: Request help: apisix-etcd Failed to bind volumes: timed out waiting for the condition
Posted by GitBox <gi...@apache.org>.
JohnnyQianP commented on issue #272:
URL: https://github.com/apache/apisix-helm-chart/issues/272#issuecomment-1107377784
In the pod, etcd has an error, what's the meaning？
```
# kubectl describe pod my-apisix-etcd-0 -n ingress-apisix
Name: my-apisix-etcd-0
Namespace: ingress-apisix
Priority: 0
Node: node2/192.168.126.102
Start Time: Sat, 23 Apr 2022 13:48:55 +0800
Labels: app.kubernetes.io/instance=my-apisix
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=etcd
controller-revision-hash=my-apisix-etcd-5b96d6bfb4
helm.sh/chart=etcd-7.0.4
statefulset.kubernetes.io/pod-name=my-apisix-etcd-0
Annotations: checksum/token-secret: 971aed8b8aa5756f775f81a27b2e5d07dae029e5ed787033e6703db5be62b008
Status: Running
IP: 10.244.2.37
IPs:
IP: 10.244.2.37
Controlled By: StatefulSet/my-apisix-etcd
Containers:
etcd:
Container ID: docker://068633dec99a85c0e2710d8a6385449262284110e4c8ee2f5138dfd52594fc40
Image: docker.io/bitnami/etcd:3.4.18-debian-10-r14
Image ID: docker-pullable://bitnami/etcd@sha256:ce0f7334f0b31f341c0bcea4b9cec6fb1e37837400fd16c257f9f6e84e35192b
Ports: 2379/TCP, 2380/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 23 Apr 2022 13:54:31 +0800
Finished: Sat, 23 Apr 2022 13:55:31 +0800
Ready: False
Restart Count: 4
Liveness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: my-apisix-etcd-0 (v1:metadata.name)
MY_STS_NAME: my-apisix-etcd
ETCDCTL_API: 3
ETCD_ON_K8S: yes
ETCD_START_FROM_SNAPSHOT: no
ETCD_DISASTER_RECOVERY: no
ETCD_NAME: $(MY_POD_NAME)
ETCD_DATA_DIR: /bitnami/etcd/data
ETCD_LOG_LEVEL: info
ALLOW_NONE_AUTHENTICATION: yes
ETCD_AUTH_TOKEN: jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
ETCD_ADVERTISE_CLIENT_URLS: http://$(MY_POD_NAME).my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://my-apisix-etcd.ingress-apisix.svc.cluster.local:2379
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://$(MY_POD_NAME).my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER: my-apisix-etcd-0=http://my-apisix-etcd-0.my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,my-apisix-etcd-1=http://my-apisix-etcd-1.my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,my-apisix-etcd-2=http://my-apisix-etcd-2.my-apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_CLUSTER_DOMAIN: my-apisix-etcd-headless.ingress-apisix.svc.cluster.local
Mounts:
/bitnami/etcd from data (rw)
/opt/bitnami/etcd/certs/token/ from etcd-jwt-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-2756j (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-my-apisix-etcd-0
ReadOnly: false
etcd-jwt-token:
Type: Secret (a volume populated by a Secret)
SecretName: my-apisix-etcd-jwt-token
Optional: false
default-token-2756j:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-2756j
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned ingress-apisix/my-apisix-etcd-0 to node2
Warning Unhealthy 5m54s kubelet, node2 Readiness probe failed: Terminated
etcd 05:49:57.13 ERROR ==> Unhealthy endpoint!
Warning Unhealthy 3m34s kubelet, node2 Readiness probe errored: rpc error: code = Unknown desc = container not running (e97a1ee83cc7d48411f94abcca346a968bee4d64d3d6c31d09b6ab05e99fe02c)
Warning Unhealthy 2m5s kubelet, node2 Readiness probe errored: rpc error: code = Unknown desc = container not running (47fd452ade0d048f45e71bfe378f4e73f8857a0ddfdd7b25791a0b144607b645)
Warning BackOff 91s (x8 over 4m51s) kubelet, node2 Back-off restarting failed container
Normal Created 80s (x5 over 6m55s) kubelet, node2 Created container etcd
Normal Started 80s (x5 over 6m55s) kubelet, node2 Started container etcd
Normal Pulled 80s (x5 over 6m55s) kubelet, node2 Container image "docker.io/bitnami/etcd:3.4.18-debian-10-r14" already present on machine
```
etcd pod error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org