You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/12/13 06:57:44 UTC
[GitHub] [apisix-helm-chart] SeibertronSS opened a new issue, #414: request help: etcd-0 CrashLoopBackOff
SeibertronSS opened a new issue, #414:
URL: https://github.com/apache/apisix-helm-chart/issues/414
### Issue description
I installed apisix-ingress using helm, but the status of pod etcd-0 is always Completed or CrashLoopBackOff. At present, the functions of apisix can be used normally. What caused the error to occur ?
```
NAME↑ PF READY RESTARTS STATUS CPU MEM %CPU/R %CPU/L %MEM/R %MEM/L IP NODE AGE │
│ apisix-dbd9d7c57-659f6 ● 1/1 0 Running 0 0 n/a n/a n/a n/a xx.xx.x.xxx xx.xx.x.xxx 13d │
│ apisix-etcd-0 ● 0/1 0 CrashLoopBackOff 0 0 n/a n/a n/a n/a xx.xx.xx.xx xx.xx.x.xxx 11m │
│ apisix-etcd-1 ● 1/1 0 Running 0 0 n/a n/a n/a n/a xx.xx.xx.xx xx.xx.x.xxx 12m │
│ apisix-etcd-2 ● 1/1 0 Running 0 0 n/a n/a n/a n/a xx.xx.xx.xx xx.xx.x.xxx 14m │
│ apisix-ingress-controller-6c694bcdc5-f97fz ● 1/1 0 Running 0 0 n/a n/a n/a n/a xx.xx.x.xxx xx.xx.x.xxx 13d
```
The log of etcd-0 is
```
etcd 07:57:55.03
etcd 07:57:55.03 Welcome to the Bitnami etcd container
etcd 07:57:55.03 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 07:57:55.03 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 07:57:55.03
etcd 07:57:55.04 INFO ==> ** Starting etcd setup **
etcd 07:57:55.05 INFO ==> Validating settings in ETCD_* env vars..
etcd 07:57:55.05 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 07:57:55.05 INFO ==> Initializing etcd
etcd 07:57:55.05 INFO ==> Generating etcd config file using env variables
etcd 07:57:55.07 INFO ==> Detected data from previous deployments
etcd 07:58:00.73 INFO ==> Member ID wasn't properly stored, the member will try to join the cluster by it's own
etcd 07:58:00.73 INFO ==> ** etcd setup finished! **
etcd 07:58:00.75 INFO ==> ** Starting etcd **
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379"}
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTH_TOKEN","variable-value":"jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m"}
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"existing"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"apisix-etcd-0"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ROOT_PASSWORD="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=apisix-etcd-headless.ingress-apisix.svc.cluster"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"warn","ts":"2022-12-12T07:58:00.776Z","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"new_member_envs","data-dir":"/bitnami/etcd/data"}
{"level":"info","ts":"2022-12-12T07:58:00.776Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"info","ts":"2022-12-12T07:58:00.776Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2022-12-12T07:58:00.776Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2022-12-12T07:58:00.784Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"08407ff76","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":32,"max-cpu-available":32,"member-initialized":true,"name":"apisix-etcd-0","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","http://apisix-etcd.ingress-apisix.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host
-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"existing","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2022-12-12T07:58:00.811Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/bitnami/etcd/data/member/snap/db","took":"5.213367ms"}
{"level":"info","ts":"2022-12-12T07:58:00.833Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2022-12-12T07:58:00.842Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"7018e2742d24ba53","local-member-id":"e1719b29f8ef586f","commit-index":3}
{"level":"info","ts":"2022-12-12T07:58:00.842Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f switched to configuration voters=()"}
{"level":"info","ts":"2022-12-12T07:58:00.842Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f became follower at term 67"}
{"level":"info","ts":"2022-12-12T07:58:00.842Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft e1719b29f8ef586f [peers: [], term: 67, commit: 3, applied: 0, lastindex: 3, lastterm: 1]"}
{"level":"info","ts":"2022-12-12T07:58:00.858Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2022-12-12T07:58:00.862Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2022-12-12T07:58:00.867Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"e1719b29f8ef586f","local-server-version":"3.5.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2022-12-12T07:58:00.867Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2022-12-12T07:58:00.868Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f switched to configuration voters=(11966697873334980856)"}
{"level":"info","ts":"2022-12-12T07:58:00.869Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"7018e2742d24ba53","local-member-id":"e1719b29f8ef586f","added-peer-id":"a6124050428dd0f8","added-peer-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2022-12-12T07:58:00.869Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.869Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"e1719b29f8ef586f","initial-advertise-peer-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","http://apisix-etcd.ingress-apisix.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"[::]:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"[::]:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.871Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8","remote-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"warn","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/http.go:426","msg":"ignored streaming request; ID mismatch","local-member-id":"e1719b29f8ef586f","remote-peer-id-stream-handler":"e1719b29f8ef586f","remote-peer-id-header":"e07f9c220823b232","remote-peer-id-from":"a6124050428dd0f8","cluster-id":"7018e2742d24ba53"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84","remote-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"warn","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"e1719b29f8ef586f","remote-peer-id-stream-handler":"e1719b29f8ef586f","remote-peer-id-from":"de632cc7e8a00e84","cluster-id":"7018e2742d24ba53"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"warn","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/http.go:426","msg":"ignored streaming request; ID mismatch","local-member-id":"e1719b29f8ef586f","remote-peer-id-stream-handler":"e1719b29f8ef586f","remote-peer-id-header":"e07f9c220823b232","remote-peer-id-from":"a6124050428dd0f8","cluster-id":"7018e2742d24ba53"}
{"level":"warn","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"e1719b29f8ef586f","remote-peer-id-stream-handler":"e1719b29f8ef586f","remote-peer-id-from":"de632cc7e8a00e84","cluster-id":"7018e2742d24ba53"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f switched to configuration voters=(11966697873334980856 16024701136227274372)"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: e1719b29f8ef586f elected leader a6124050428dd0f8 at term 67"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"7018e2742d24ba53","local-member-id":"e1719b29f8ef586f","added-peer-id":"de632cc7e8a00e84","added-peer-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84","remote-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f switched to configuration voters=(11966697873334980856 16024701136227274372 16244935935474489455)"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"7018e2742d24ba53","local-member-id":"e1719b29f8ef586f","added-peer-id":"e1719b29f8ef586f","added-peer-peer-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"warn","ts":"2022-12-12T07:58:00.875Z","caller":"etcdserver/server.go:1150","msg":"server error","error":"the member has been permanently removed from the cluster"}
{"level":"warn","ts":"2022-12-12T07:58:00.875Z","caller":"etcdserver/server.go:1151","msg":"data-dir used by this member must be removed"}
{"level":"warn","ts":"2022-12-12T07:58:00.875Z","caller":"etcdserver/server.go:2053","msg":"stopped publish because server is stopped","local-member-id":"e1719b29f8ef586f","local-member-attributes":"{Name:apisix-etcd-0 ClientURLs:[http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379 http://apisix-etcd.ingress-apisix.svc.cluster.local:2379]}","publish-timeout":"7s","error":"etcdserver: server stopped"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"de632cc7e8a00e84"}
{"level":"warn","ts":"2022-12-12T07:58:00.884Z","caller":"rafthttp/http.go:145","msg":"failed to process Raft message","local-member-id":"e1719b29f8ef586f","error":"raft: stopped"}
{"level":"info","ts":"2022-12-12T07:58:00.886Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2022-12-12T07:58:00.886Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
```
### Environment
- your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
- your Kubernetes cluster version (output of kubectl version):
- if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] niuguy commented on issue #414: request help: etcd-0 CrashLoopBackOff
Posted by GitBox <gi...@apache.org>.
niuguy commented on issue #414:
URL: https://github.com/apache/apisix-helm-chart/issues/414#issuecomment-1351073905
I got a similar issue, etcd keeps crashing, this is my error log
```
2022-12-14 05:04:04 | etcd 05:04:04.00
-- | --
| | 2022-12-14 05:04:04 | etcd 05:04:04.00 Welcome to the Bitnami etcd container
| | 2022-12-14 05:04:04 | etcd 05:04:04.00 Subscribe to project updates by watching https://github.com/bitnami/containers
| | 2022-12-14 05:04:04 | etcd 05:04:04.00 Submit issues and feature requests at https://github.com/bitnami/containers/issues
| | 2022-12-14 05:04:04 | etcd 05:04:04.00
| | 2022-12-14 05:04:04 | etcd 05:04:04.01 INFO ==> ** Starting etcd setup **
| | 2022-12-14 05:04:04 | etcd 05:04:04.02 INFO ==> Validating settings in ETCD_* env vars..
| | 2022-12-14 05:04:04 | etcd 05:04:04.02 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
| | 2022-12-14 05:04:04 | etcd 05:04:04.03 INFO ==> Initializing etcd
| | 2022-12-14 05:04:04 | etcd 05:04:04.03 INFO ==> Generating etcd config file using env variables
| | 2022-12-14 05:04:04 | etcd 05:04:04.04 INFO ==> Detected data from previous deployments
| | 2022-12-14 05:04:04 | etcd 05:04:04.16 INFO ==> Updating member in existing cluster
| | 2022-12-14 05:04:04 | {"level":"warn","ts":"2022-12-14T05:04:04.211Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00031ca80/apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
| | 2022-12-14 05:04:04 | Error: etcdserver: member not found
```
I was able to make it work by first deleting the PersistenceVolumeClaim of the failed etcd pod, then restarting the pod. But I don't know why it worked and failed previously. If someone can help explain 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tao12345666333 commented on issue #414: request help: etcd-0 CrashLoopBackOff
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #414:
URL: https://github.com/apache/apisix-helm-chart/issues/414#issuecomment-1351357101
you could search in issue first https://github.com/apache/apisix-helm-chart/issues?q=is%3Aissue+is%3Aopen+CrashLoopBackOff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org