You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Delfino Gustavo <Gu...@zf.com> on 2019/04/21 17:25:15 UTC
CouchDB replication crashing
Hello, I have had two CouchDB servers replicating a couple of DBs for several months. One server is local and the other is in remote location. Both are running CouchDB 2.2.0 under Windows 7. Replication is failing and I know I can reactivate it by restarting the remote server but I would like to understand if there is anything I can do to prevent this situation in the first place.
In previous CouchDB versions, I used to have this problem where the replication seems to be disconnected from the network while the DB continues to be accessible. It had been working very well under 2.2.0. I wonder if moving up to 2.3 would help.
A representative segment of the log file is found below.
Regards,
Gustavo Delfino
[error] 2019-04-21T12:38:21.805000Z couchdb@localhost <0.28098.851> -------- Replicator, request PUT to "http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false" failed due to error {code,500}
[error] 2019-04-21T12:38:24.020000Z couchdb@localhost <0.8055.852> -------- Replicator, request PUT to "http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false" failed due to error {code,500}
[error] 2019-04-21T12:38:28.497000Z couchdb@localhost <0.12289.852> -------- Replicator, request PUT to "http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false" failed due to error {code,500}
[error] 2019-04-21T12:38:51.819000Z couchdb@localhost <0.11004.852> -------- Replicator, request GET to "http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-211?revs=true&open_revs=%5B%221-4c79aa0880ec04131b40c06855b36ff4%22%5D&latest=true" failed due to error {error,req_timedout}
[error] 2019-04-21T12:38:51.819000Z couchdb@localhost <0.9866.852> -------- Replicator, request GET to "http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-15?atts_since=%5B%222-597b62c56ad573a21a63e62c1117856e%22%5D&revs=true&open_revs=%5B%223-5f29e1678a58784ee2e8aee46c57b29d%22%5D&latest=true" failed due to error sel_conn_closed
[error] 2019-04-21T12:38:52.240000Z couchdb@localhost <0.21284.852> -------- Replicator, request PUT to "http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false" failed due to error {code,500}
[error] 2019-04-21T12:39:01.678000Z couchdb@localhost <0.19310.852> -------- Replicator, request POST to "http://nearbyhost.com:5984/jobhandlersims/_bulk_docs" failed due to error {code,500}
[error] 2019-04-21T12:39:01.678000Z couchdb@localhost <0.9042.852> -------- gen_server <0.9042.852> terminated with reason: {process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
last msg: {'EXIT',<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
state: [{main_pid,<0.9322.852>},{loop,<0.6467.852>},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_fetch,nil},{batch_size,268141}]
[error] 2019-04-21T12:39:01.678000Z couchdb@localhost <0.9042.852> -------- CRASH REPORT Process (<0.9042.852>) with 1 neighbors exited with reason: {process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}} at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [], links: [<0.9322.852>,<0.6467.852>], dictionary: [{last_stats_report,{1555,850299,901000}}], trap_exit: true, status: running, heap_size: 6772, stack_size: 27, reductions: 393291
[error] 2019-04-21T12:39:18.916000Z couchdb@localhost <0.9687.852> -------- Replicator, request POST to "http://nearbyhost.com:5984/jobhandlersims/_bulk_docs" failed due to error {code,500}
[error] 2019-04-21T12:39:18.916000Z couchdb@localhost <0.12947.851> -------- gen_server <0.12947.851> terminated with reason: {process_died,<0.9687.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
last msg: {'EXIT',<0.9687.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
state: [{main_pid,<0.9322.852>},{loop,<0.15756.851>},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_fetch,nil},{batch_size,383624}]
[error] 2019-04-21T12:39:18.916000Z couchdb@localhost <0.12947.851> -------- CRASH REPORT Process (<0.12947.851>) with 1 neighbors exited with reason: {process_died,<0.9687.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}} at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [], links: [<0.9322.852>,<0.15756.851>], dictionary: [{last_stats_report,{1555,850299,839000}}], trap_exit: true, status: running, heap_size: 6772, stack_size: 27, reductions: 552772
[error] 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> -------- Replicator, request POST to "http://nearbyhost.com:5984/jobhandlersims/_bulk_docs" failed due to error {code,500}
[error] 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> -------- gen_server <0.10607.852> terminated with reason: {http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}} at couch_replicator_httpc:report_error/4(line:363) <= couch_replicator_httpc:send_req/3(line:76) <= couch_replicator_worker:flush_docs/2(line:493) <= couch_replicator_worker:maybe_flush_docs/3(line:459) <= couch_replicator_worker:maybe_flush_docs/2(line:436) <= couch_replicator_worker:handle_call/3(line:124) <= gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
last msg: {batch_doc,{doc,<<"4bb0aa8269a879fdc95107973872dcfb-424">>,{3,[<<157,204,36,86,54,61,219,128,174,147,218,239,156,55,31,23>>,<<61,194,111,234,124,76,158,204,215,47,185,48,158,105,104,215>>,<<203,140,2,100,128,7,203,236,171,131,32,29,132,106,170,51>>]}, (...erlang json removed...) ,[],false,[]}}
state: [{main_pid,<0.9322.852>},{loop,<0.1165.852>},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,20},{pending_fetch,{{<0.1165.852>,#Ref<0.0.23068677.99464>},{<<"4bb0aa8269a879fdc95107973872dcfb-174">>,[{3,<<139,102,22,196,97,193,111,166,119,37,58,206,57,73,182,130>>}],[{2,<<132,199,224,202,205,103,3,105,55,29,255,196,200,63,170,152>>}]}}},{batch_size,523788}]
[error] 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> -------- CRASH REPORT Process (<0.10607.852>) with 15 neighbors exited with reason: {http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}} at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [{'EXIT',<0.24052.852>,normal},{'$gen_call',{<0.20083.852>,...},...},...], links: [<0.22451.852>,<0.24660.852>,<0.25927.852>,<0.27245.852>,...], dictionary: [{last_stats_report,{1555,850301,711000}},{ibrowse_stream_status,...}], trap_exit: true, status: running, heap_size: 28690, stack_size: 27, reductions: 744275
[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> -------- Worker <0.9042.852> died with reason: {process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> -------- Replication `6ccccb02a7483deae5c1c434bce9b376+continuous+create_target` (`http://farawayhost.com:5984/jobhandlersims/` -> `http://nearbyhost.com:5984/jobhandlersims/`) failed: {worker_died,<0.9042.852>,
{process_died,<0.19310.852>,
{http_request_failed,"POST",
"http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",
{error,{code,500}}}}}
[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> -------- gen_server {couch_replicator_scheduler_job,{[54,99,99,99,99,98,48,50,97,55,52,56,51,100,101,97,101,53,99,49,99,52,51,52,98,99,101,57,98,51,55,54],[43,99,111,110,116,105,110,117,111,117,115,43,99,114,101,97,116,101,95,116,97,114,103,101,116]}} terminated with reason: {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
last msg: {'EXIT',<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
state: [{rep_id,{"6ccccb02a7483deae5c1c434bce9b376","+continuous+create_target"}},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{db_name,<<"shards/40000000-5fffffff/_replicator.1526394302">>},{doc_id,<<"DE_to_US_for_jobhandlersims">>},{options,[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{create_target,true},{http_connections,20},{retries,5},{selector,{[{<<"$or">>,[{[{<<"type">>,{[{<<"$ne">>,<<"results">>}]}}]},{[{<<"views">>,{[{<<"$gte">>,<<>>}]}}]},{[{<<"type">>,<<"results">>},{<<"requester_location">>,<<"US">>}]}]}]}},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}]},{session_id,<<"436547a63366ea84a38d8aeb36dd2d88">>},{start_seq,{0,<<"3194311-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsZqDDTMwsUizMDIg1bAEkGH1cMNUDCGGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQD5VMZk">>}},{source_seq,<<"3229426-g1AAAAFreJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8riYHtnzQedUkKQDLJHqqUTQ6fUgeQ0nioUokz-JQmgJTWQ5WaLcSjNI8FSDI0ACmg6vkg5R0PCSpfAFG-H6icNUKdoPIDEOX3QconXyCo_AFEOdjtdSVZAOiudpk">>},{committed_seq,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{current_through_seq,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{highest_seq_done,{8,<<"3204835-g1AAAAI3eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge3T91ygGLu5qWFqimEyNj14TEpSAJJJ9iDDkhhYf17Cp9QBpDQebi-3MNheAxOLNAsjA1LtTQAZVg-xl01zKR6leSxAkqEBSAFVz4fY3egIttsy0Sgx2dKCRLshBi6AGLgfbCCrVxPYQKO0REsLM2OyDDwAMfA-xMB2C0ismBslpyaakmXgA4iB0OAuSgcbaJKUkmxkmYhNaxYAmZez-A">>}}]
[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> -------- CRASH REPORT Process (<0.9322.852>) with 3 neighbors exited with reason: {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}} at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors: [couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [{'EXIT',<0.12947.851>,{process_died,<0.9687.852>,...}},...], links: [<0.162.852>,<0.5422.852>,<0.10686.852>,<0.25534.850>,...], dictionary: [{pending_count_state,{{1555,850367,995000},9348}},{task_status_update,...},...], trap_exit: true, status: running, heap_size: 10958, stack_size: 27, reductions: 63152
[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.10686.852> -------- gen_server <0.10686.852> terminated with reason: {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
last msg: {'EXIT',<0.9322.852>,{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}}
state: [{main_pid,<0.9322.852>},{loop,<0.6805.852>},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,1},{pending_fetch,nil},{batch_size,351185}]
[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.10686.852> -------- CRASH REPORT Process (<0.10686.852>) with 2 neighbors exited with reason: {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}} at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...], messages: [], links: [<0.6805.852>,<0.20253.851>], dictionary: [{last_stats_report,{1555,850299,839000}}], trap_exit: true, status: running, heap_size: 2586, stack_size: 27, reductions: 496765
[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.385.0> -------- Supervisor couch_replicator_scheduler_sup had child undefined started with {couch_replicator_scheduler_job,start_link,undefined} at <0.9322.852> exit with reason {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}} in context child_terminated
[error] 2019-04-21T12:40:13.313000Z couchdb@localhost <0.23845.851> 7ec5e9ec3e rexi_server: from: couchdb@localhost(<0.28700.851>) mfa
: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/
fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:40:13.423000Z couchdb@localhost <0.19545.851> 7ec5e9ec3e rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:40:13.438000Z couchdb@localhost <0.26362.851> 7ec5e9ec3e rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:40:13.438000Z couchdb@localhost <0.25940.851> 7ec5e9ec3e rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:40:13.438000Z couchdb@localhost <0.25878.851> 7ec5e9ec3e rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:40:13.438000Z couchdb@localhost <0.25863.851> 7ec5e9ec3e rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:40:13.438000Z couchdb@localhost <0.28372.851> 7ec5e9ec3e rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:40:13.438000Z couchdb@localhost <0.29759.851> 7ec5e9ec3e rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:43:23.323000Z couchdb@localhost <0.16895.850> ed368f9947 rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}]
[error] 2019-04-21T12:43:23.339000Z couchdb@localhost <0.3976.852> ed368f9947 rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}]}]
[error] 2019-04-21T12:43:23.339000Z couchdb@localhost <0.3667.852> ed368f9947 rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: fabric_rpc:changes/3 exit:timeout [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}]}]