You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/04/21 10:27:14 UTC
[GitHub] [apisix] hexiang1219 opened a new issue, #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
hexiang1219 opened a new issue, #6902:
URL: https://github.com/apache/apisix/issues/6902
### Description
At present, there are about 27 apisix nodes in the environment, each of which has 48 process synchronization, etc. The current node reports a large number of errors as follows:
[error] 95896#95896: *16160233305 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/routes, context: ngx.timer
[error] 95898#95898: *16158933459 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/consumers, context: ngx.timer
[error] 95926#95926: *16155389245 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/ssl, context: ngx.timer
[error] 95938#95938: *16478531139 [lua] config_etcd.lua:569: failed to fetch data from etcd: connection reset by peer, etcd key: /apisix/routes, context: ngx.timer
10.10.1.1 This etcd node has the highest number of connections, and now the average is about 15k. The other two nodes have about 700 connections.
### Environment
apisix version :2.9:
OS (3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux)
etcd version:3.5:
OS (3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux)
My config.yaml body:
apisix:
etcd:
host:
- "http://10.10.1.1:2379"
- "http://10.10.1.2:2379"
- "http://10.10.1.3:2379"
etcd:
ETCD_QUOTA_BACKEND_BYTES="8589934592"
ETCD_AUTO_COMPACTION_MODE="revision"
ETCD_AUTO_COMPACTION_RETENTION="1"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1987169116
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] hexiang1219 commented on issue #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
Posted by GitBox <gi...@apache.org>.
hexiang1219 commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1105963245
The node compression key is 0, and the data synchronization delay is large.
The log of this node has some errors and warnings:
etcd[229645]: {"level":"panic","ts":"2022-04-22T11:08:41.714+0800","caller":"mvcc/watchable_store.go:414","msg":"failed to unmarshal mvccpb.KeyValue","error":"proto: KeyValue: illegal tag 0 (wire type 0)","stacktrace":"go.etcd.io/etcd/server/v3/mvcc.kvsToEvents\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/mvcc/watchable_store.go:414\ngo.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncWatchers\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/mvcc/watchable_store.go:359\ngo.etcd.io/etcd/server/v3/mvcc.(*watchableStore).syncWatchersLoop\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/mvcc/watchable_store.go:222"}
{"level":"warn","ts":"2022-01-04T19:19:15.215+0800","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"12.728µs","request":"header:<ID:12351823333449688949 > compaction:<revision:4872183 > ","response":"","error":"mvcc: required revision is a future revision"}
{"level":"warn","ts":"2022-01-04T19:19:15.215+0800","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"12.728µs","request":"header:<ID:12351823333449688949 > compaction:<revision:4872183 > ","response":"","error":"mvcc: required revision is a future revision"}
{"level":"warn","ts":"2021-12-29T02:04:12.810+0800","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"100.74585ms","expected-duration":"100ms","prefix":"","request":"header:<ID:12351823333449401808 > compaction:<revision:4610817 > ","response":"","error":"mvcc: required revision is a future revision"}
{"level":"warn","ts":"2021-12-22T15:26:34.846+0800","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"a31ba4ef82825f60","error":"failed to dial a31ba4ef82825f60 on stream MsgApp v2 (peer a31ba4ef82825f60 failed to find local node 5913db87427e034)"}
{"level":"warn","ts":"2021-12-22T15:26:34.845+0800","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"5913db87427e034","remote-peer-id":"a31ba4ef82825f60","error":"EOF"}
[core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1:2379 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] coffeebe4code commented on issue #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
Posted by "coffeebe4code (via GitHub)" <gi...@apache.org>.
coffeebe4code commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1464268019
Same issue here. ETCD is definitely crashing during this time. You can see my error [here](https://github.com/etcd-io/etcd/issues/15448).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1962316913
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1106037547
you need to fix your ETCD cluster problems first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer [apisix]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
URL: https://github.com/apache/apisix/issues/6902
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1105387161
look like that your ETCD cluster is unhealthy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1105889669
Any monitoring data about ETCD cluster?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Lewisyixin commented on issue #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
Posted by GitBox <gi...@apache.org>.
Lewisyixin commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1112210539
@hexiang1219 HI , have you fix this issue? I faced the same issue but found nothing error about etcd cluster..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #6902: [error] 66637#66637: *1935502945 [lua] config_etcd.lua:569: failed to fetch data from etcd: closed, etcd key: /apisix/upstreams, context: ngx.timer
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #6902:
URL: https://github.com/apache/apisix/issues/6902#issuecomment-1112794476
> @hexiang1219 HI , have you fix this issue? I faced the same issue but found nothing error about etcd cluster..
Also, check the connectivity between Apache APISIX and ETCD.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org