[GitHub] mariocasciaro commented on issue #1106: Error opening views: badmatch/ badarg, ets member mem3_openers

[GitBox <gi...@apache.org>]

mariocasciaro commented on issue #1106: Error opening views: badmatch/ badarg, ets member mem3_openers
URL: https://github.com/apache/couchdb/issues/1106#issuecomment-437930176
 
 
   Trying to add some more information here since I have similar errors in my log (actually a lot of them):
   
   ```
   [error] 2018-11-12T15:03:04.488080Z couchdb@xxx.xxx emulator -------- Error in process <0.1582.0> on node 'couchdb@xxx.xxx' with exit value:
   {badarg,[{ets,member,[mem3_openers,<<"xxx">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,91}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}
   
   [error] 2018-11-12T15:03:04.488130Z couchdb@xxx.xxx <0.244.0> -------- Error opening view group `rel` from database `shards/00000000-ffffffff/xxx.1534776614`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"xxx">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,91}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,337}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,257}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,228}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',4,[{file,"src/couch_compaction_daemon.erl"},{line,142}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,315}]},{couch_util,fold_files2,6,[{file,"src/couch_util.erl"},{line,108}]},{couch_util,fold_files2,6,[{file,"src/couch_util.erl"},{line,113}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,305}]}]}}
   ```
   
   Usually those errors are preceded by lines like these:
   
   ```
   [notice] 2018-11-12T15:02:58.436921Z couchdb@xxx.xxx <0.276.0> -------- rexi_server_mon : cluster unstable
   [notice] 2018-11-12T15:02:58.436962Z couchdb@xxx.xxx <0.279.0> -------- rexi_server_mon : cluster unstable
   [notice] 2018-11-12T15:02:58.437193Z couchdb@xxx.xxx <0.275.0> -------- rexi_server : cluster unstable
   [notice] 2018-11-12T15:02:58.437287Z couchdb@xxx.xxx <0.278.0> -------- rexi_buffer : cluster unstable
   [notice] 2018-11-12T15:02:58.469005Z couchdb@xxx.xxx <0.276.0> -------- rexi_server_mon : cluster unstable
   [notice] 2018-11-12T15:02:58.469046Z couchdb@xxx.xxx <0.279.0> -------- rexi_server_mon : cluster unstable
   [notice] 2018-11-12T15:02:58.469310Z couchdb@xxx.xxx <0.275.0> -------- rexi_server : cluster unstable
   [notice] 2018-11-12T15:02:58.469357Z couchdb@xxx.xxx <0.278.0> -------- rexi_buffer : cluster unstable
   ```
   
   I also noticed that the stat `number of rexi_DOWN messages handled` is very high, it increases of about 1000 every 10 minutes.
   
   Other info:
   ```
   q=1
   n=3
   
   Databases: around 200, most of them < 1mb, a few up to 40mb, total size 200mb
   Load on the database: max 100 requests per minute
   Load on machines: very low, CPU avg 2%, Max memory usage 40%
   Machines specs: 1 vCpu, 2GB ram
   I can consistently see those errors on all the 3 machines in the cluster and all have about the same number of 'rexi_DOWN messages handled'.
   ```
   
   The same application and database configuration (but with different data) is running on another environment, actually with higher load and many more databases and I can't see all those error messages.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services