You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2019/08/27 13:16:03 UTC
[GitHub] [couchdb] mariocasciaro opened a new issue #2147: High number of
rexi_DOWN messages
mariocasciaro opened a new issue #2147: High number of rexi_DOWN messages
URL: https://github.com/apache/couchdb/issues/2147
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
I have multiple clusters with `q=1` and `n=3`, with different data but same configuration. On one cluster when I query `_node/_local/_stats` I get a very high value in the parameter `number of rexi_DOWN messages handled`. The other clusters have 0 or a very low value for the same parameter.
On the surface, the affected cluster does not appear to have any functionality issues. However given the critical nature of the data it stores, I want to be sure that these errors are not a symptom of some deeper problem which might compromise the server and the data.
Extract from `_node/_local/_stats`:
```
{
"rexi": {
"buffered": {
"value": 1,
"type": "counter",
"desc": "number of rexi messages buffered"
},
"down": {
"value": 425565,
"type": "counter",
"desc": "number of rexi_DOWN messages handled"
},
"dropped": {
"value": 0,
"type": "counter",
"desc": "number of rexi messages dropped from buffers"
}
}
}
```
In the logs I noticed some `cluster unstable` messages which may or may not be related to the issue.
```
[notice] 2019-08-21T23:19:10.858745Z couchdb@xxx.xx <0.275.0> -------- rexi_server : started servers
[notice] 2019-08-21T23:19:10.867696Z couchdb@xxx.xx <0.279.0> -------- rexi_buffer : started servers
[info] 2019-08-21T23:19:10.868037Z couchdb@xxx.xx <0.9.0> -------- Application rexi started on node 'couchdb@xxx.xx'
[notice] 2019-08-21T23:19:10.988247Z couchdb@xxx.xx <0.276.0> -------- rexi_server_mon : cluster unstable
[notice] 2019-08-21T23:19:10.988294Z couchdb@xxx.xx <0.280.0> -------- rexi_server_mon : cluster unstable
[notice] 2019-08-21T23:19:10.988535Z couchdb@xxx.xx <0.275.0> -------- rexi_server : cluster unstable
[notice] 2019-08-21T23:19:10.988615Z couchdb@xxx.xx <0.279.0> -------- rexi_buffer : cluster unstable
[notice] 2019-08-21T23:19:11.035312Z couchdb@xxx.xx <0.276.0> -------- rexi_server_mon : cluster unstable
[notice] 2019-08-21T23:19:11.035349Z couchdb@xxx.xx <0.280.0> -------- rexi_server_mon : cluster unstable
[notice] 2019-08-21T23:19:11.035465Z couchdb@xxx.xx <0.275.0> -------- rexi_server : cluster unstable
[notice] 2019-08-21T23:19:11.035503Z couchdb@xxx.xx <0.279.0> -------- rexi_buffer : cluster unstable
```
Also, in the logs I have some errors related to `rexi_server`:
```
[error] 2019-08-26T10:12:36.704836Z couchdb@xxx.xx <0.27053.209> -------- rexi_server: from: couchdb@couchdb-eu-3.d4h.live(<14388.27013.1517>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,265}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,205}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,462}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,526}]},{couch_bt_engine,include_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,1074}]},{couch_bt_engine,skip_deleted,4,[{file,"src/couch_bt_engine.erl"},{line,1069}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]}]
[error] 2019-08-26T10:12:36.704897Z couchdb@xxx.xx <0.3634.210> -------- rexi_server: from: couchdb@couchdb-eu-1.d4h.live(<14387.921.191>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,"src/rexi.erl"},{line,265}]},{rexi,stream2,3,[{file,"src/rexi.erl"},{line,205}]},{fabric_rpc,view_cb,2,[{file,"src/fabric_rpc.erl"},{line,462}]},{couch_mrview,map_fold,3,[{file,"src/couch_mrview.erl"},{line,526}]},{couch_bt_engine,include_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,1074}]},{couch_bt_engine,skip_deleted,4,[{file,"src/couch_bt_engine.erl"},{line,1069}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]}]
```
## Your Environment
* CouchDB Version used: 2.3.0
* Operating System and version: Ubuntu 18.04
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services