[GitHub] [couchdb] oldrich-svec commented on issue #3571: Compaction dies constantly after a certain amount of documents

[GitBox <gi...@apache.org>]

oldrich-svec commented on issue #3571:
URL: https://github.com/apache/couchdb/issues/3571#issuecomment-845800181


   We have a similar issue (Ubuntu Server 20.04, Docker version of CouchDB 3.1.1).
   
   We have a 30GB database which is being replicated from an another machine. Looking at the files I can see that the database files take someting like 100GB + another ca. 30GB for the compaction files.
   
   The compaction starts but always dies before it finishes. So the database gets never compacted and the compaction files are hanging there forever.
   
   Some logs:
   
   ```
   couchdb-backup-service-01 | [info] 2021-05-21T08:27:57.466850Z nonode@nohost <0.222.0> -------- db shards/80000000-ffffffff/yoda_filesystem.1614838752 died with reason {{badarg,[{erlang,monitor,[process,{main,'clouseau@127.0.0.1'}],[]},{ioq,submit_request,2,[{file,"src/ioq.erl"},{line,187}]},{ioq,maybe_submit_request,1,[{file,"src/ioq.erl"},{line,150}]},{ioq,handle_info,2,[{file,"src/ioq.erl"},{line,123}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,616}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,686}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{gen_server,call,[ioq,{request,<0.24352.1>,{append_bin,[<<0,0,32,0>>,[<<6,66,160,149,155,150,189,119,66,50,121,52,14,190,9,143...}}
   couchdb-backup-service-01 | [warning] 2021-05-21T08:27:57.467050Z nonode@nohost <0.428.0> -------- exit for compaction of ["shards/00000000-7fffffff/yoda_filesystem.1614838752"]: {{badarg,[{erlang,monitor,[process,{main,'clouseau@127.0.0.1'}],[]},{ioq,submit_request,2,[{file,"src/ioq.erl"},{line,187}]},{ioq,maybe_submit_request,1,[{file,"src/ioq.erl"},{line,150}]},{ioq,handle_info,2,[{file,"src/ioq.erl"},{line,123}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,616}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,686}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{gen_server,call,[ioq,{request,<0.22588.1>,{pread_iolist,40340132370},compaction,<0.22589.1>,undefined},infinity]}}
   couchdb-backup-service-01 | [info] 2021-05-21T08:27:57.467190Z nonode@nohost <0.222.0> -------- db shards/00000000-7fffffff/yoda_filesystem.1614838752 died with reason {{badarg,[{erlang,monitor,[process,{main,'clouseau@127.0.0.1'}],[]},{ioq,submit_request,2,[{file,"src/ioq.erl"},{line,187}]},{ioq,maybe_submit_request,1,[{file,"src/ioq.erl"},{line,150}]},{ioq,handle_info,2,[{file,"src/ioq.erl"},{line,123}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,616}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,686}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{gen_server,call,[ioq,{request,<0.22588.1>,{pread_iolist,40340132370},compaction,<0.22589.1>,undefined},infinity]}}
   
   couchdb-backup-service-01 | [info] 2021-05-21T08:27:57.468479Z nonode@nohost <0.28448.1> -------- Starting compaction for db "shards/80000000-ffffffff/yoda_filesystem.1614838752" at 125776
   couchdb-backup-service-01 | [notice] 2021-05-21T08:27:57.468708Z nonode@nohost <0.428.0> -------- ratio_dbs: Started compaction for shards/80000000-ffffffff/yoda_filesystem.1614838752
   couchdb-backup-service-01 | [error] 2021-05-21T08:27:57.469504Z nonode@nohost <0.22508.1> -------- gen_server ioq terminated with reason: bad argument in call to erlang:monitor(process, {main,'clouseau@127.0.0.1'}) at ioq:submit_request/2(line:187) <= ioq:maybe_submit_request/1(line:150) <= ioq:handle_info/2(line:123) <= gen_server:try_dispatch/4(line:616) <= gen_server:handle_msg/6(line:686) <= proc_lib:init_p_do_apply/3(line:247)
   couchdb-backup-service-01 |   last msg: timeout
   couchdb-backup-service-01 |      state: {state,10,0.01,{[{request,{main,'clouseau@127.0.0.1'},{open,<0.28430.1>,<<"shards/80000000-ffffffff/yodadev_features.1614849634/de6fcb4cc39fbd3e0c1d621d441e9057">>,<<"standard">>},other,{<0.28430.1>,#Ref<0.2791382983.2
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org