You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/05/12 06:54:50 UTC
[GitHub] [apisix-helm-chart] zhangyihong opened a new issue, #282: etcd的三个POD之一，一直报错重启
zhangyihong opened a new issue, #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282
k8s环境：
1. AWS宁夏区 eks
```
kubectl version --short
Client Version: v1.19.6-eks-49a6c0
Server Version: v1.20.15-eks-0d102a7
```
2. chart版本：
```
helm list -n ingress-apisix
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
apisix ingress-apisix 1 2022-05-12 14:37:31.160379 +0800 CST deployed apisix-0.9.2 2.13.1
```
安装方式：
```
helm install apisix apisix/apisix \
--set gateway.type=LoadBalancer \
--set ingress-controller.enabled=true \
--namespace ingress-apisix \
--set ingress-controller.config.apisix.serviceNamespace=ingress-apisix
```
POD,svc 详情：
```
kubectl get po -n ingress-apisix -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apisix-8fd7755b8-gssjm 1/1 Running 0 10m 172.19.110.47 ip-172-19-103-234.cn-northwest-1.compute.internal <none> <none>
apisix-etcd-0 1/1 Running 0 10m 172.19.97.230 ip-172-19-108-124.cn-northwest-1.compute.internal <none> <none>
apisix-etcd-1 0/1 CrashLoopBackOff 6 10m 172.19.125.194 ip-172-19-103-234.cn-northwest-1.compute.internal <none> <none>
apisix-etcd-2 1/1 Running 0 10m 172.19.161.188 ip-172-19-164-161.cn-northwest-1.compute.internal <none> <none>
apisix-ingress-controller-699bf76956-4x78m 1/1 Running 0 10m 172.19.109.163 ip-172-19-103-234.cn-northwest-1.compute.internal <none> <none>
kubectl get svc -n ingress-apisix
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
apisix-admin ClusterIP 10.100.192.112 <none> 9180/TCP 10m
apisix-etcd ClusterIP 10.100.73.13 <none> 2379/TCP,2380/TCP 10m
apisix-etcd-headless ClusterIP None <none> 2379/TCP,2380/TCP 10m
apisix-gateway LoadBalancer 10.100.73.47 xxxx-xxxxx.cn-northwest-1.elb.amazonaws.com.cn 80:32049/TCP 10m
apisix-ingress-controller ClusterIP 10.100.158.3 <none> 80/TCP 10m
```
问题：
上面可以看到apisix-etcd-1 一直重启，查看该pod的日志如下：
```
kubectl logs apisix-etcd-1 -n ingress-apisix
etcd 06:48:47.33
etcd 06:48:47.33 Welcome to the Bitnami etcd container
etcd 06:48:47.34 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 06:48:47.34 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 06:48:47.34
etcd 06:48:47.34 INFO ==> ** Starting etcd setup **
etcd 06:48:47.36 INFO ==> Validating settings in ETCD_* env vars..
etcd 06:48:47.37 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 06:48:47.38 INFO ==> Initializing etcd
etcd 06:48:47.38 INFO ==> Generating etcd config file using env variables
etcd 06:48:47.40 INFO ==> Detected data from previous deployments
etcd 06:48:47.50 INFO ==> Updating member in existing cluster
{"level":"warn","ts":"2022-05-12T06:48:47.594Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-cfa2c90d-8acd-4633-86ea-aaf91758985e/apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
Error: etcdserver: member not found
```
```
kubectl describe pod apisix-etcd-1 -n ingress-apisix
Name: apisix-etcd-1
Namespace: ingress-apisix
Priority: 0
Node: ip-172-19-103-234.cn-northwest-1.compute.internal/172.19.103.234
Start Time: Thu, 12 May 2022 14:50:49 +0800
Labels: app.kubernetes.io/instance=apisix
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=etcd
controller-revision-hash=apisix-etcd-6b644fb7bf
helm.sh/chart=etcd-7.0.4
statefulset.kubernetes.io/pod-name=apisix-etcd-1
Annotations: checksum/token-secret: c1e6eac2a55b08a1b379e89dd5133d17f5e112ef9ebab1429a62ef704f42f72b
kubernetes.io/psp: eks.privileged
Status: Running
IP: 172.19.125.146
IPs:
IP: 172.19.125.146
Controlled By: StatefulSet/apisix-etcd
Containers:
etcd:
Container ID: docker://b69e9d4673842170cc8e3306f61314e9801c8a3a3de03a2e1957ea0c32f9f20c
Image: docker.io/bitnami/etcd:3.4.18-debian-10-r14
Image ID: docker-pullable://bitnami/etcd@sha256:ce0f7334f0b31f341c0bcea4b9cec6fb1e37837400fd16c257f9f6e84e35192b
Ports: 2379/TCP, 2380/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 12 May 2022 14:52:41 +0800
Finished: Thu, 12 May 2022 14:52:41 +0800
Ready: False
Restart Count: 4
Liveness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh] delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: apisix-etcd-1 (v1:metadata.name)
MY_STS_NAME: apisix-etcd
ETCDCTL_API: 3
ETCD_ON_K8S: yes
ETCD_START_FROM_SNAPSHOT: no
ETCD_DISASTER_RECOVERY: no
ETCD_NAME: $(MY_POD_NAME)
ETCD_DATA_DIR: /bitnami/etcd/data
ETCD_LOG_LEVEL: info
ALLOW_NONE_AUTHENTICATION: yes
ETCD_AUTH_TOKEN: jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
ETCD_ADVERTISE_CLIENT_URLS: http://$(MY_POD_NAME).apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS: http://$(MY_POD_NAME).apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER: apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_CLUSTER_DOMAIN: apisix-etcd-headless.ingress-apisix.svc.cluster.local
Mounts:
/bitnami/etcd from data (rw)
/opt/bitnami/etcd/certs/token/ from etcd-jwt-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rdhrk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-apisix-etcd-1
ReadOnly: false
etcd-jwt-token:
Type: Secret (a volume populated by a Secret)
SecretName: apisix-etcd-jwt-token
Optional: false
default-token-rdhrk:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rdhrk
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m14s default-scheduler Successfully assigned ingress-apisix/apisix-etcd-1 to ip-172-19-103-234.cn-northwest-1.compute.internal
Normal Pulled 22s (x5 over 116s) kubelet Container image "docker.io/bitnami/etcd:3.4.18-debian-10-r14" already present on machine
Normal Created 22s (x5 over 116s) kubelet Created container etcd
Normal Started 22s (x5 over 116s) kubelet Started container etcd
Warning BackOff 22s (x10 over 115s) kubelet Back-off restarting failed container
```
删除该pod，让它重建一次，还是不能解决问题：
```
kubectl delete pod apisix-etcd-1 -n ingress-apisix
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] amondnet commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
amondnet commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1125921447
```
etcd 10:24:37.50
etcd 10:24:37.50 Welcome to the Bitnami etcd container
etcd 10:24:37.50 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 10:24:37.50 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 10:24:37.51
etcd 10:24:37.51 INFO ==> ** Starting etcd setup **
etcd 10:24:37.55 INFO ==> Validating settings in ETCD_* env vars..
etcd 10:24:37.55 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 10:24:37.56 INFO ==> Initializing etcd
etcd 10:24:37.56 INFO ==> Generating etcd config file using env variables
etcd 10:24:37.58 INFO ==> There is no data from previous deployments
etcd 10:24:37.58 INFO ==> Bootstrapping a new cluster
etcd 10:24:42.60 INFO ==> Obtaining cluster member ID
etcd 10:24:42.61 INFO ==> Starting etcd in background
```
```
etcd 10:30:03.62 Welcome to the Bitnami etcd container
etcd 10:30:03.63 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 10:30:03.63 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 10:30:03.63
etcd 10:30:03.63 INFO ==> ** Starting etcd setup **
etcd 10:30:03.67 INFO ==> Validating settings in ETCD_* env vars..
etcd 10:30:03.67 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 10:30:03.68 INFO ==> Initializing etcd
etcd 10:30:03.68 INFO ==> Generating etcd config file using env variables
etcd 10:30:03.69 INFO ==> Detected data from previous deployments
etcd 10:30:03.79 INFO ==> Updating member in existing cluster
Error: bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID in Hex
```
Solution
1. set ETCD_INITIAL_CLUSTER_STATE : existing
2. delete pvc
3. restart pod
```
etcd 10:45:53.52
etcd 10:45:53.52 Welcome to the Bitnami etcd container
etcd 10:45:53.52 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 10:45:53.53 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 10:45:53.53
etcd 10:45:53.53 INFO ==> ** Starting etcd setup **
etcd 10:45:53.55 INFO ==> Validating settings in ETCD_* env vars..
etcd 10:45:53.56 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 10:45:53.56 INFO ==> Initializing etcd
etcd 10:45:53.56 INFO ==> Generating etcd config file using env variables
etcd 10:45:53.57 INFO ==> There is no data from previous deployments
etcd 10:45:53.57 INFO ==> Adding new member to existing cluster
etcd 10:45:53.66 INFO ==> Obtaining cluster member ID
etcd 10:45:53.66 INFO ==> Starting etcd in background
etcd 10:45:56.73 INFO ==> Stored member ID: 17a7caef1fb55e37
etcd 10:45:56.74 INFO ==> Stopping etcd
Terminated
etcd 10:45:57.75 INFO ==> ** etcd setup finished! **
etcd 10:45:57.77 INFO ==> ** Starting etcd **
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] amondnet commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
amondnet commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124849246
Is this related to https://github.com/bitnami/charts/issues/9865?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] A-little-bit-of-data commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
A-little-bit-of-data commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1278399279
> infrastructure
Yes, all etcd nodes are not started normally, the error is the same as above（ Headless service domain does not have an IP per initial member in the cluster）, I use bitnami's apisix chart package, this etcd is included in this chart package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] gxthrj commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
gxthrj commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124666384
> Can you list the detailed operation steps?
>
> ```
> kubectl exec -it apisix-etcd-0 -n ingress-apisix -- bash
> I have no name!@apisix-etcd-0:/opt/bitnami/etcd$
> ```
>
> next step ?
Try about this
`etcdctl --endpoints "http://x.x.x.x:2379" member list` and `etcdctl --endpoints "http://x.x.x.x:2379" cluster-health`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] amondnet commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
amondnet commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124856797
```bash
I have no name!@apisix-etcd-1:/opt/bitnami/etcd$ etcdctl --endpoints "http://apisix-etcd.ingress-apisix.svc.cluster.local:2379" member list
```
```bash
46fdf952e73505c2, started, apisix-etcd-0, http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380, http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379, false
47324e080c98137d, started, apisix-etcd-1, http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380, http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379, false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tokers commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1125816132
@zhangyihong I'm curious about the error logs printed by the abnormal ETCD instance. Do you still reserve any related logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tao12345666333 commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124651816
hi, can you run check etcd member list? And try to use remove current member
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] zhangyihong closed issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
zhangyihong closed issue #282: etcd的三个POD之一，一直报错重启
URL: https://github.com/apache/apisix-helm-chart/issues/282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] A-little-bit-of-data commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
A-little-bit-of-data commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1277147696
> ```
> etcd 10:24:37.50
> etcd 10:24:37.50 Welcome to the Bitnami etcd container
> etcd 10:24:37.50 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
> etcd 10:24:37.50 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
> etcd 10:24:37.51
> etcd 10:24:37.51 INFO ==> ** Starting etcd setup **
> etcd 10:24:37.55 INFO ==> Validating settings in ETCD_* env vars..
> etcd 10:24:37.55 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
> etcd 10:24:37.56 INFO ==> Initializing etcd
> etcd 10:24:37.56 INFO ==> Generating etcd config file using env variables
> etcd 10:24:37.58 INFO ==> There is no data from previous deployments
> etcd 10:24:37.58 INFO ==> Bootstrapping a new cluster
> etcd 10:24:42.60 INFO ==> Obtaining cluster member ID
> etcd 10:24:42.61 INFO ==> Starting etcd in background
> ```
>
> ```
> etcd 10:30:03.62 Welcome to the Bitnami etcd container
> etcd 10:30:03.63 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
> etcd 10:30:03.63 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
> etcd 10:30:03.63
> etcd 10:30:03.63 INFO ==> ** Starting etcd setup **
> etcd 10:30:03.67 INFO ==> Validating settings in ETCD_* env vars..
> etcd 10:30:03.67 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
> etcd 10:30:03.68 INFO ==> Initializing etcd
> etcd 10:30:03.68 INFO ==> Generating etcd config file using env variables
> etcd 10:30:03.69 INFO ==> Detected data from previous deployments
> etcd 10:30:03.79 INFO ==> Updating member in existing cluster
> Error: bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID in Hex
> ```
>
> Solution
>
> 1. set ETCD_INITIAL_CLUSTER_STATE : existing
> 2. delete pvc
> 3. restart pod
>
> and I use etcd image `amond/etcd:3.4.18-debian-10-r30` https://hub.docker.com/layers/213541121/amond/etcd/3.4.18-debian-10-r30/images/sha256-c8575e6e2fffcf5fb7492c270be4771b008ee24f64ed820bfbbe0a09d290d4ca?context=repo
>
> [amondnet/bitnami-docker-etcd@94c7f67](https://github.com/amondnet/bitnami-docker-etcd/commit/94c7f673cc71cefb3deeebe0b4769f8ec9541d72)
>
> [bitnami/bitnami-docker-etcd#39 (comment)](https://github.com/bitnami/bitnami-docker-etcd/issues/39#issuecomment-1069322973)
>
> ```
> etcd 10:45:53.52
> etcd 10:45:53.52 Welcome to the Bitnami etcd container
> etcd 10:45:53.52 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
> etcd 10:45:53.53 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
> etcd 10:45:53.53
> etcd 10:45:53.53 INFO ==> ** Starting etcd setup **
> etcd 10:45:53.55 INFO ==> Validating settings in ETCD_* env vars..
> etcd 10:45:53.56 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
> etcd 10:45:53.56 INFO ==> Initializing etcd
> etcd 10:45:53.56 INFO ==> Generating etcd config file using env variables
> etcd 10:45:53.57 INFO ==> There is no data from previous deployments
> etcd 10:45:53.57 INFO ==> Adding new member to existing cluster
> etcd 10:45:53.66 INFO ==> Obtaining cluster member ID
> etcd 10:45:53.66 INFO ==> Starting etcd in background
> etcd 10:45:56.73 INFO ==> Stored member ID: 17a7caef1fb55e37
> etcd 10:45:56.74 INFO ==> Stopping etcd
> Terminated
> etcd 10:45:57.75 INFO ==> ** etcd setup finished! **
>
> etcd 10:45:57.77 INFO ==> ** Starting etcd **
> ```
I did the same operation as you, but my etcd service still can't get up. What's the reason?
The version I am using is 3.4.18-debian-10-r14 of bitnami
The error log of etcd is as follows
etcd 07:21:09.99
etcd 07:21:10.00 Welcome to the Bitnami etcd container
etcd 07:21:10.00 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 07:21:10.00 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 07:21:10.01
etcd 07:21:10.01 INFO ==> ** Starting etcd setup **
etcd 07:21:10.04 INFO ==> Validating settings in ETCD_* env vars..
etcd 07:21:10.04 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 07:21:10.05 INFO ==> Initializing etcd
etcd 07:21:10.06 INFO ==> Generating etcd config file using env variables
etcd 07:21:10.08 INFO ==> There is no data from previous deployments
etcd 07:21:10.08 INFO ==> Bootstrapping a new cluster
etcd 07:22:10.23 ERROR ==> Headless service domain does not have an IP per initial member in the cluster
> ```
> etcd 10:24:37.50
> etcd 10:24:37.50 Welcome to the Bitnami etcd container
> etcd 10:24:37.50 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
> etcd 10:24:37.50 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
> etcd 10:24:37.51
> etcd 10:24:37.51 INFO ==> ** Starting etcd setup **
> etcd 10:24:37.55 INFO ==> Validating settings in ETCD_* env vars..
> etcd 10:24:37.55 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
> etcd 10:24:37.56 INFO ==> Initializing etcd
> etcd 10:24:37.56 INFO ==> Generating etcd config file using env variables
> etcd 10:24:37.58 INFO ==> There is no data from previous deployments
> etcd 10:24:37.58 INFO ==> Bootstrapping a new cluster
> etcd 10:24:42.60 INFO ==> Obtaining cluster member ID
> etcd 10:24:42.61 INFO ==> Starting etcd in background
> ```
>
> ```
> etcd 10:30:03.62 Welcome to the Bitnami etcd container
> etcd 10:30:03.63 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
> etcd 10:30:03.63 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
> etcd 10:30:03.63
> etcd 10:30:03.63 INFO ==> ** Starting etcd setup **
> etcd 10:30:03.67 INFO ==> Validating settings in ETCD_* env vars..
> etcd 10:30:03.67 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
> etcd 10:30:03.68 INFO ==> Initializing etcd
> etcd 10:30:03.68 INFO ==> Generating etcd config file using env variables
> etcd 10:30:03.69 INFO ==> Detected data from previous deployments
> etcd 10:30:03.79 INFO ==> Updating member in existing cluster
> Error: bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID in Hex
> ```
>
> Solution
>
> 1. set ETCD_INITIAL_CLUSTER_STATE : existing
> 2. delete pvc
> 3. restart pod
>
> and I use etcd image `amond/etcd:3.4.18-debian-10-r30` https://hub.docker.com/layers/213541121/amond/etcd/3.4.18-debian-10-r30/images/sha256-c8575e6e2fffcf5fb7492c270be4771b008ee24f64ed820bfbbe0a09d290d4ca?context=repo
>
> [amondnet/bitnami-docker-etcd@94c7f67](https://github.com/amondnet/bitnami-docker-etcd/commit/94c7f673cc71cefb3deeebe0b4769f8ec9541d72)
>
> [bitnami/bitnami-docker-etcd#39 (comment)](https://github.com/bitnami/bitnami-docker-etcd/issues/39#issuecomment-1069322973)
>
> ```
> etcd 10:45:53.52
> etcd 10:45:53.52 Welcome to the Bitnami etcd container
> etcd 10:45:53.52 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
> etcd 10:45:53.53 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
> etcd 10:45:53.53
> etcd 10:45:53.53 INFO ==> ** Starting etcd setup **
> etcd 10:45:53.55 INFO ==> Validating settings in ETCD_* env vars..
> etcd 10:45:53.56 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
> etcd 10:45:53.56 INFO ==> Initializing etcd
> etcd 10:45:53.56 INFO ==> Generating etcd config file using env variables
> etcd 10:45:53.57 INFO ==> There is no data from previous deployments
> etcd 10:45:53.57 INFO ==> Adding new member to existing cluster
> etcd 10:45:53.66 INFO ==> Obtaining cluster member ID
> etcd 10:45:53.66 INFO ==> Starting etcd in background
> etcd 10:45:56.73 INFO ==> Stored member ID: 17a7caef1fb55e37
> etcd 10:45:56.74 INFO ==> Stopping etcd
> Terminated
> etcd 10:45:57.75 INFO ==> ** etcd setup finished! **
>
> etcd 10:45:57.77 INFO ==> ** Starting etcd **
> ```
I did the same operation as you, but my etcd service still can't get up. What's the reason?
The version I am using is 3.4.18-debian-10-r14 of bitnami
The error log of etcd is as follows
etcd 07:21:09.99
etcd 07:21:10.00 Welcome to the Bitnami etcd container
etcd 07:21:10.00 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 07:21:10.00 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 07:21:10.01
etcd 07:21:10.01 INFO ==> ** Starting etcd setup **
etcd 07:21:10.04 INFO ==> Validating settings in ETCD_* env vars..
etcd 07:21:10.04 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 07:21:10.05 INFO ==> Initializing etcd
etcd 07:21:10.06 INFO ==> Generating etcd config file using env variables
etcd 07:21:10.08 INFO ==> There is no data from previous deployments
etcd 07:21:10.08 INFO ==> Bootstrapping a new cluster
etcd 07:22:10.23 ERROR ==> Headless service domain does not have an IP per initial member in the cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tao12345666333 commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1277742111
> etcd 07:22:10.23 ERROR ==> Headless service domain does not have an IP per initial member in the cluster
You can try to check your infrastructure. Or are all etcd nodes unavailable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] etcd的三个POD之一，一直报错重启 [apisix-helm-chart]
Posted by "finaldie (via GitHub)" <gi...@apache.org>.
finaldie commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1945374521
Luckily I found this old issue thread here; as of today, 2024-02-14, the exact problem (etcdserver: member not found) still happens to me.
etcd: `bitnami/etcd:3.5.12-debian-11-r1`
While in the prod environment, we cannot uninstall `etcd` helm package simply as all other services depend on it, we may want to minimize the downtime as small as possible.
Share what I did here:
1. Delete PVC from one terminal (it will hang there as the pod is still there)
2. Open another terminal, `kubectl delete pod -n <namespace> <pod-name>`
Then, the pod will be deleted and corresponding pvc/pv will be deleted together, after that a new pod will be automatically created, and a new pvc/pv will be created as well.
Thank you all for the discussion and solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] zhangyihong commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
zhangyihong commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124660779
> Is this related to [etcd-io/etcd#13724](https://github.com/etcd-io/etcd/issues/13724) ?
我升级etcd to : 3.5.4-debian-10-r17，还是一样的问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] amondnet commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
amondnet commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1165405104
https://github.com/bitnami/bitnami-docker-etcd/pull/42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] zyhnesmr commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by "zyhnesmr (via GitHub)" <gi...@apache.org>.
zyhnesmr commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1578650788
> > > infrastructure
> >
> >
> > Yes, all etcd nodes are not started normally, the error is the same as above（ Headless service domain does not have an IP per initial member in the cluster）, I use bitnami's apisix chart package, this etcd is included in this chart package
>
> 我遇到了和你一样的问题，所有的pod报相同的错误，但是我在为所有的etcd准备好pv后，确认所有的pvc已经绑定，删除 pod apisix-etcd-0 ，待其重启之后，问题就解决了。
感谢 我遇到相同的问题 已解决 respect🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] zhangyihong commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
zhangyihong commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124658740
Can you list the detailed operation steps?
```
kubectl exec -it apisix-etcd-0 -n ingress-apisix -- bash
```
next step ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] zhangyihong commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
zhangyihong commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124683786
```
kubectl exec -it apisix-etcd-0 -n ingress-apisix -- bash
I have no name!@apisix-etcd-0:/opt/bitnami/etcd$ etcdctl --endpoints "http://apisix-etcd.ingress-apisix.svc.cluster.local:2379" member list
7b13756c7f9b6c15, started, apisix-etcd-2, http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380, http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379, false
7c23e3a78699f17b, started, apisix-etcd-0, http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380, http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379, false
```
```
etcdctl --endpoints "http://apisix-etcd.ingress-apisix.svc.cluster.local:2379" endpoint health
http://apisix-etcd.ingress-apisix.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 13.799211ms
I have no name!@apisix-etcd-0:/opt/bitnami/etcd$
I have no name!@apisix-etcd-0:/opt/bitnami/etcd$
I have no name!@apisix-etcd-0:/opt/bitnami/etcd$ etcdctl --endpoints "http://apisix-etcd.ingress-apisix.svc.cluster.local:2379" cluster-health
Error: unknown command "cluster-health" for "etcdctl"
Run 'etcdctl --help' for usage.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] gxthrj commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
gxthrj commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124614779
Is this related to https://github.com/etcd-io/etcd/issues/13724 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] gxthrj commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
gxthrj commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1124613373
cc @tao12345666333 PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] zhangyihong commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
zhangyihong commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1125877086
> @zhangyihong我很好奇ETCD实例异常打印的错误日志。您还保留任何相关日志吗？
>
之前出现问题的时候，故障的etcd实例只有一个警告日志，如下：
{"level":"warn","ts":"2022-05-12T06:48:47.594Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-cfa2c90d-8acd-4633-86ea-aaf91758985e/apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
Error: etcdserver: member not found
```
kubectl logs apisix-etcd-1 -n ingress-apisix
etcd 06:48:47.33
etcd 06:48:47.33 Welcome to the Bitnami etcd container
etcd 06:48:47.34 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 06:48:47.34 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 06:48:47.34
etcd 06:48:47.34 INFO ==> ** Starting etcd setup **
etcd 06:48:47.36 INFO ==> Validating settings in ETCD_* env vars..
etcd 06:48:47.37 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 06:48:47.38 INFO ==> Initializing etcd
etcd 06:48:47.38 INFO ==> Generating etcd config file using env variables
etcd 06:48:47.40 INFO ==> Detected data from previous deployments
etcd 06:48:47.50 INFO ==> Updating member in existing cluster
{"level":"warn","ts":"2022-05-12T06:48:47.594Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-cfa2c90d-8acd-4633-86ea-aaf91758985e/apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
Error: etcdserver: member not found
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] GMsure commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by "GMsure (via GitHub)" <gi...@apache.org>.
GMsure commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1422077344
> > infrastructure
>
> Yes, all etcd nodes are not started normally, the error is the same as above（ Headless service domain does not have an IP per initial member in the cluster）, I use bitnami's apisix chart package, this etcd is included in this chart package
我遇到了和你一样的问题，所有的pod报相同的错误，但是我在为所有的etcd准备好pv后，确认所有的pvc已经绑定，删除 pod apisix-etcd-0 ，待其重启之后，问题就解决了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tao12345666333 commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1278515342
So please check if your infrastructure is healthy and look at the logs of each pod to make sure you find the real cause
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] zhangyihong commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
zhangyihong commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1125614163
我尝试删除etcd 使用的pv,pvc后，再部署一次，ETCD三个pod都不再发生错误导致重启了。
具体过程如下：
1. 卸载apisix
```
helm uninstall apisix -n ingress-apisix
```
2. 删除etcd使用的pv
```
#查询etcd的pv,pvc:
kubectl get pvc,pv -n ingress-apisix |grep apisix
persistentvolumeclaim/data-apisix-etcd-0 Bound pvc-03976dfd-a1fb-4fcd-ba14-4d4eb050d6fc 8Gi RWO gp2 23d
persistentvolumeclaim/data-apisix-etcd-1 Bound pvc-7ba580bc-d6fa-406a-a0b8-6aaff2ba3c01 8Gi RWO gp2 23d
persistentvolumeclaim/data-apisix-etcd-2 Bound pvc-fed68916-d3ac-4b4a-a663-c0685d0e5884 8Gi RWO gp2 23d
persistentvolume/pvc-03976dfd-a1fb-4fcd-ba14-4d4eb050d6fc 8Gi RWO Delete Bound ingress-apisix/data-apisix-etcd-0 gp2 23d
persistentvolume/pvc-7ba580bc-d6fa-406a-a0b8-6aaff2ba3c01 8Gi RWO Delete Bound ingress-apisix/data-apisix-etcd-1 gp2 23d
persistentvolume/pvc-fed68916-d3ac-4b4a-a663-c0685d0e5884 8Gi RWO Delete Bound ingress-apisix/data-apisix-etcd-2 gp2 23d
#删除所有pvc(相应的pv也会删除):
kubectl delete pvc data-apisix-etcd-0 -n ingress-apisix
persistentvolumeclaim "data-apisix-etcd-0" deleted
kubectl delete pvc data-apisix-etcd-1 -n ingress-apisix
persistentvolumeclaim "data-apisix-etcd-1" deleted
kubectl delete pvc data-apisix-etcd-2 -n ingress-apisix
persistentvolumeclaim "data-apisix-etcd-2" deleted
```
3. 重写安装apisix
```
#更新repo
helm repo update
#安装
helm install apisix apisix/apisix \
--set gateway.type=LoadBalancer \
--set ingress-controller.enabled=true \
--namespace ingress-apisix \
--set ingress-controller.config.apisix.serviceNamespace=ingress-apisix
```
4. 查询安装的版本：
发现已经有更新了 apisix-0.9.2 ---> apisix-0.9.3
```
helm list -n ingress-apisix
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
apisix ingress-apisix 1 2022-05-13 10:38:41.243799 +0800 CST deployed apisix-0.9.3 2.13.1
```
5. 检查pod的运行状态：
```
kubectl get po -n ingress-apisix
NAME READY STATUS RESTARTS AGE
apisix-8fd7755b8-fckm6 1/1 Running 0 7m56s
apisix-etcd-0 1/1 Running 0 7m56s
apisix-etcd-1 1/1 Running 0 7m56s
apisix-etcd-2 1/1 Running 0 7m56s
apisix-ingress-controller-999c5bf97-hh7mh 1/1 Running 0 7m56s
```
etcd的三个pod 都运行正常，不再发生重启！
6. 检查etcd 的成员列表也有3个了：
```
kubectl exec -it apisix-etcd-0 -n ingress-apisix -- bash
I have no name!@apisix-etcd-0:/opt/bitnami/etcd$
I have no name!@apisix-etcd-0:/opt/bitnami/etcd$ etcdctl --endpoints "http://apisix-etcd.ingress-apisix.svc.cluster.local:2379" member list
39e3f227d2d33a6d, started, apisix-etcd-2, http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380, http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379, false
47324e080c98137d, started, apisix-etcd-1, http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380, http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379, false
e2571e968b89c849, started, apisix-etcd-0, http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380, http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379, false
```
7. 不清楚是切底删除etcd的pv还是升级apisix-0.9.2 ---> apisix-0.9.3后，导致问题得到解决。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tao12345666333 commented on issue #282: etcd的三个POD之一，一直报错重启
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #282:
URL: https://github.com/apache/apisix-helm-chart/issues/282#issuecomment-1125702436
Great! Apparently this is due to you reinstalling etcd.
You can see the changes from 0.9.2 to 0.9.3 not related to etcd.
https://github.com/apache/apisix-helm-chart/compare/apisix-0.9.2...apisix-0.9.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org