You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Carlos Alonso <ca...@cabify.com> on 2017/08/02 12:40:39 UTC
Re: Understanding a couple replication errors
Hi Joan, many thanks for your reply
Why would the timeout and closed connections happen? I see it quite often
on the logs TBH but also, a slightly different one is a timeout against the
cluster itself. What may this mean?
[error] 2017-08-02T12:38:27.321697Z couchdb@couch-2 <0.7114.568> --------
Replicator, request PUT to "http://admin:*****@
127.0.0.1:5984/my_db/35e78fe9d421677c77dea34522de2b99?new_edits=false"
failed due to error {error,{conn_failed,{error,etimedout}}}
Regards
On Mon, Jul 31, 2017 at 5:06 PM Joan Touzet <wo...@apache.org> wrote:
> Hi Carlos,
>
> This issue (the {{nocatch,{mp_parser_died,noproc}}) is recently
> identified and patched!
>
> https://github.com/apache/couchdb/issues/574
>
> The other error just looks like a disconnection. You may want to
> wait (just a couple more days!) for 2.1.0 which has all sorts of
> new replication features and try again.
>
> -Joan
>
> ----- Original Message -----
> From: "Carlos Alonso" <ca...@cabify.com>
> To: "user" <us...@couchdb.apache.org>
> Sent: Monday, July 31, 2017 9:58:09 AM
> Subject: Re: Understanding a couple replication errors
>
> On top of those, this one happens from time to time as well:
>
> [error] 2017-07-31T13:41:50.258423Z couchdb@couch-2 emulator --------
> Error
> in process <0.16620.2434> on node 'couchdb@couch-2' with exit value:
>
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},...
>
> [error] 2017-07-31T13:41:50.260569Z couchdb@couch-2 <0.32053.2433>
> --------
> Replicator, request PUT to "http://admin:*****@
> 127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false"
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
> [{couch_att,'-foldl/4-fun-0-',3,
> [{file,"src/couch_att.erl"},{line,591}]},
> {couch_att,fold_streamed_data,4,
> [{file,"src/couch_att.erl"},{line,642}]},
> {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
> {couch_httpd_multipart,atts_to_mp,4,
> [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
> {gen_server,call,
> [<0.11924.2435>,
> {send_req,
> {{url,
> "
>
> http://admin:password@127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false
> ",
> "127.0.0.1",5984,"admin","password",
>
> "/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false",
> http,ipv4_address},
> [{"Accept","application/json"},
> {"Content-Length",32845},
> {"Content-Type",
> "multipart/related;
> boundary=\"46b37c9ba185ac67bf24ee40bcc38828\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
> put,
> {#Fun<couch_replicator_api_wrap.11.79455979>,
>
>
> {<<"{\"_id\":\"cba0aa16e347ac7d6b09732032828704\",\"_rev\":\"8-b90e6bd0b6281f303e587f4b9538a0af\",...\"_revisions\":{\"start\":8,\"ids\":[\"b90e6bd0b6281f303e587f4b9538a0af\",\"df66a380ea1258243bfd8a33506a18b4\",\"4cd932d1b71cebc4a55f91331d97f902\",\"a4dabd1dbf2b1de8987119a65cb44131\",\"82cc46c88ca768ff092de95c338bc0e9\",\"b4a7562ac8a831f46a194e97ea33e527\",\"d4fd4ddb998e4022e0335f609ed1ef50\",\"118cae6fc6864b3c48f788e78705c8cb\"]},\"_attachments\":{\"FF01-0020941.pdf\":{\"content_type\":\"application/pdf\",\"revpos\":8,\"digest\":\"md5-IT8Ms2HDG9EFLvfIxYTaVw==\",\"length\":21888,\"follows\":true},\"FF01-0020941.xml\":{\"content_type\":\"application/xml\",\"revpos\":8,\"digest\":\"md5-KJIXHDQbcJ3xeLaOUh11cw==\",\"length\":8526,\"follows\":true}}}">>,
> [{att,<<"FF01-0020941.pdf">>,<<"application/pdf">>,
> 21888,21888,
>
> <<33,63,12,179,97,195,27,209,5,46,247,200,197,132,
> 218,87>>,
> 8,
> {follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
> identity},
>
> {att,<<"FF01-0020941.xml">>,<<"application/xml">>,8526,
> 8526,
>
> <<40,146,23,28,52,27,112,157,241,120,182,142,82,29,
> 117,115>>,
> 8,
> {follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
> identity}],
> <<"46b37c9ba185ac67bf24ee40bcc38828">>,32845}},
> [{response_format,binary},
> {inactivity_timeout,30000},
> {socket_options,[{keepalive,true},{nodelay,false}]}],
> infinity}},
> infinity]}}}}
>
>
> Thanks!
>
>
>
> On Mon, Jul 31, 2017 at 11:52 AM Carlos Alonso <ca...@cabify.com>
> wrote:
>
> > Hi all!!
> >
> > I've been observing logs for a while and trying to understand why a few
> > errors happen while (pull) replication is happening and also trying to
> > understand the consequences.
> >
> > ** connection_closed*
> >
> > Log:
> > ```
> > [error] <0.15968.1946> -------- Replicator, request GET to
> "https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2210613426-g1AAAAOCeJyd0r1OwzAQB3CrRYIHYC8FCSZS2_kwEUM7svURiu1zVKqQSlUZYKFP0FeAV2DsBE_BCm-Suj4UKqRYIctZsv66n053OSHkaNoFcgpKzxdmBIrRQEt1lz1SygKdzx9AFsugMMvcZjuSqF5ZlrNpV5Hj8-t7-3eYZJyDzoD0qyZhTY8_lPBR6sRWNay0_pvTmJA8ScS-xhppPPJqo502qbSbwGlUUwNXrMVsiVe73WnPlXbx5DSQXBihW8xWF3NacWArWdnHgi8oXn7ifGHEeST_L_rPBMVXFN9RHORO5FowIeKm4m_KeywIfiD4heD6zIFpbCAL4xYrrIvtid8o_qxxvHGiiVKTQqM1zrad2hfx%22&timeout=10000"
> > failed due to error connection_closed
> > ```
> >
> > Cause: This one happens from time to time. It looks like the source is
> > closing connection, but why?
> > Consequences: None. I think replicator just retries the task.
> >
> > * *changes_reader_died*
> >
> > Log:
> > ```
> > [error] <0.15968.1946> -------- Replicator, request GET to
> "https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000"
> > failed due to error connection_closed
> >
> > [error] <0.13952.1946> -------- ChangesReader process died with reason:
> > {http_request_failed,"GET","
> > https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed
> > <https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000%22,%7Berror,connection_closed>
> > }}
> >
> > [error] <0.13952.1946> -------- Replication
> > `363ed31e35361d024d22e0a990ca1ae5+continuous` (`https://admin:
> *****@source.ip:443/my_db/`
> > -> `http://admin:*****@127.0.0.1:5984/my_db/`
> <http://127.0.0.1:5984/my_db/>
> > <http://127.0.0.1:5984/my_db/>) failed: changes_reader_died
> >
> > [error] <0.30404.6252> -------- Replicator, request GET to
> "https://admin:
> *****@source.ip:443/my_db/b666ee7ec8ed5b82ae035fe5fa6a4a2c?revs=true&open_revs=%5B%221-5a7ba8a8658bd0ab7b3656aa089aa183%22%5D&latest=true"
> > failed due to error {error,closing_on_request}
> >
> > [error] <0.352.0> -------- Error in replication
> > `363ed31e35361d024d22e0a990ca1ae5+continuous` (triggered by document
> > `28d973af08931f79fa423b54a8004b31`): changes_reader_died
> > Restarting replication in 5 seconds.
> >
> > [error] <0.13952.1946> -------- gen_server <0.13952.1946> terminated with
> > reason: changes_reader_died
> > last msg:
> > {'EXIT',<0.15968.1946>,{http_request_failed,"GET","https://admin:
> >
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed}}}
> > state:
> >
> [{data,[{"State",{rep_state,{rep,{"363ed31e35361d024d22e0a990ca1ae5","+continuous"},{httpdb,"
> https://admin:
> >
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,nil,20},{httpdb,"
> http://admin:
> > *****@127.0.0.1:5984/my_db/
> >
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,nil,20},[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{http_connections,20},{retries,10},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}],{user_ctx,null,[],undefined},db,nil,<<"28d973af08931f79fa423b54a8004b31">>,<<"shards/c0000000-dfffffff/_replicator.1501167373">>},"
> https://admin:
> > *****@source.ip:443/my_db/","http://admin:*****@127.0.0.1:5984/my_db/
> > ",{httpdb,"https://admin:
> >
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"
> http://admin:
> > *****@127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[],{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"source_last_seq">>,<<"36567936-g1...">>},{<<"replication_id_version">>,3},{<<"history">>,[{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"start_time">>,<<"Fri,
> > 28 Jul 2017 09:46:48 GMT">>},{<<"end_time">>,<<"Sun, 30 Jul 2017 02:28:59
> >
> GMT">>},{<<"start_last_seq">>,0},{<<"end_last_seq">>,<<"36567936-g1...">>},{<<"recorded_seq">>,<<"36567936-g1...">>},{<<"missing_checked">>,22991404},{<<"missing_found">>,22991404},{<<"docs_read">>,22991404},{<<"docs_written">>,22991404},{<<"doc_write_failures">>,0}]}]}]},{0,0},{46381,<<"36567936-g1...">>},{46388,<<"36574076-g1...">>},[{46389,<<"36575175-g1...">>},{46390,<<"36575938-g1...">>},{46391,<<"36576695-...">>},{46392,<<"36577391-g1...">>}],{46388,<<"36574076-g1...">>},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},"Fri,
> > 28 Jul 2017 09:46:48
> >
> GMT",<<"0">>,<<"0">>,nil,<0.16590.1946>,<0.16953.1946>,<0.15968.1946>,[<0.18469.1946>,<0.10009.1946>,<0.16617.1946>,<0.16155.1946>],[{doc_write_failures,0},{docs_read,22994904},{docs_written,22994904},{missing_checked,22994904},{missing_found,22994904}],<<"6f7aa01e879f5e1841cdb7100f40806b">>,nil,nil,nil,nil,<<"255587270-g1...">>,true,30000,db,nil}}]}]
> >
> > [error] <0.13952.1946> -------- CRASH REPORT Process (<0.13952.1946>)
> > with 2 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator,init,['Argument__1']}, ancestors:
> > [couch_replicator_job_sup,couch_replicator_sup,<0.349.0>], messages: [],
> > links: [<0.16590.1946>,<0.16953.1946>,<0.18469.1946>,<0.16617.1946>,...],
> > dictionary:
> >
> [{task_status_props,[{changes_pending,127404519},{checkpoint_interval,...},...]},...],
> > trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> > reductions: 26502159
> >
> > [error] <0.364.0> -------- Supervisor couch_replicator_job_sup had child
> > "363ed31e35361d024d22e0a990ca1ae5+continuous" started with
> > {gen_server,start_link,undefined} at <0.13952.1946> exit with reason
> > changes_reader_died in context child_terminated
> >
> > [error] <0.16617.1946> -------- gen_server <0.16617.1946> terminated with
> > reason: changes_reader_died
> > last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> > state: {state,<0.13952.1946>,<0.15004.1946>,20,{httpdb,"
> > https://admin:
> ******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:
> ******@127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.2293.6254>,<0.19041.6253>,<0.19785.6253>,<0.7470.6253>,<0.23117.6253>,<0.31026.6250>,<0.11251.6253>,<0.27821.6253>,<0.8477.6253>,<0.28980.6253>,<0.22848.6253>,<0.16549.6251>,<0.26208.6253>,<0.27365.6253>,<0.15830.6253>,<0.24330.6253>,<0.20562.6253>,<0.20869.6253>,<0.21799.6253>,<0.16311.6253>],nil,{{<0.15004.1946>,#Ref<0.0.6636.248998>},{<<"07597cb4d450c2b6ff674b675b9d5168">>,[{1,<<93,162,57,7,69,142,28,226,77,41,199,156,216,131,208,152>>}],[]}},nil,[{docs_read,199},{docs_written,25},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"b65rnw8pcm0jbt7jdpxvgmo1r\",\"_rev\":\"2-4d5283c4d125a0da73b95cc80b92abd6\
> > ",...}}
> >
> > [error] <0.16155.1946> -------- gen_server <0.16155.1946> terminated with
> > reason: changes_reader_died
> > last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> > state: {state,<0.13952.1946>,<0.21653.1946>,20,{httpdb,"
> > https://admin:
> ******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:
> ******@127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.27802.6253>,<0.27692.6253>,<0.10519.6253>,<0.5584.6253>,<0.9485.6253>,<0.2597.6253>,<0.12901.6253>,<0.21982.6253>,<0.20758.6253>,<0.12367.6253>,<0.30450.6253>,<0.15900.6253>,<0.17133.6253>,<0.22945.6253>,<0.8889.6253>,<0.20333.6253>,<0.12924.6253>,<0.32204.6252>,<0.32140.6253>,<0.5556.6253>],nil,{{<0.21653.1946>,#Ref<0.0.6636.251466>},{<<"9lcen85eu5ztrg8cs287sacoy">>,[{2,<<58,122,97,121,57,131,34,12,88,18,7,165,20,247,0,70>>}],[]}},nil,[{docs_read,436},{docs_written,57},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"1802v33hahs2ql9951vitjbih\",\"_rev\":\"5-138167bbafe87da23205b457c3918bd4\
> > ",...],...}}
> >
> > [error] <0.16617.1946> -------- CRASH REPORT Process (<0.16617.1946>)
> > with 15 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> > messages: [], links:
> > [<0.21799.6253>,<0.24330.6253>,<0.27365.6253>,<0.28980.6253>,...],
> > dictionary:
> > [{ibrowse_stream_status,init},{last_stats_report,{1501,381757,...}}],
> > trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> > reductions: 6921530028
> >
> > [error] <0.16155.1946> -------- CRASH REPORT Process (<0.16155.1946>)
> > with 15 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> > messages: [], links:
> > [<0.12924.6253>,<0.21982.6253>,<0.27692.6253>,<0.30450.6253>,...],
> > dictionary:
> > [{ibrowse_stream_status,init},{last_stats_report,{1501,381751,...}}],
> > trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> > reductions: 6914251593
> >
> > [error] <0.18469.1946> -------- gen_server <0.18469.1946> terminated with
> > reason: changes_reader_died
> > last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> > state: {state,<0.13952.1946>,<0.21935.1946>,20,{httpdb,"
> > https://admin:
> ******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:
> ******@127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.12414.6254>,<0.24987.6253>,<0.24685.6253>,<0.30475.6253>,<0.25412.6253>,<0.18441.6253>,<0.23627.6253>,<0.10529.6252>,<0.18425.6253>,<0.18994.6250>,<0.2214.6253>,<0.9544.6253>,<0.13738.6253>,<0.32462.6253>,<0.15946.6253>,<0.25304.6253>,<0.16690.6253>,<0.3468.6253>,<0.17659.6253>,<0.13537.6253>],nil,{{<0.21935.1946>,#Ref<0.0.6636.248486>},{<<"935cd70fbe4e02cb374180daa08a9fff">>,[{2,<<194,192,159,166,5,0,237,72,59,193,108,4,228,198,91,95>>}],[]}},nil,[{missing_checked,500},{missing_found,500}],nil,nil,{batch,[],0
> > }}
> >
> > [error] <0.10009.1946> -------- gen_server <0.10009.1946> terminated with
> > reason: changes_reader_died
> > last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> > state: {state,<0.13952.1946>,<0.22823.1946>,20,{httpdb,"
> > https://admin:
> ******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:
> ******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\
> > ",,...],...}}
> >
> > [error] <0.18469.1946> -------- CRASH REPORT Process (<0.18469.1946>)
> > with 15 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> > messages: [], links:
> > [<0.18441.6253>,<0.25304.6253>,<0.30475.6253>,<0.32462.6253>,...],
> > dictionary:
> > [{ibrowse_stream_status,init},{last_stats_report,{1501,381762,...}}],
> > trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> > reductions: 6918077761
> >
> > [error] <0.10009.1946> -------- CRASH REPORT Process (<0.10009.1946>)
> > with 15 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> > messages: [], links:
> > [<0.18053.6253>,<0.23938.6253>,<0.29957.6253>,<0.31079.6253>,...],
> > dictionary:
> > [{ibrowse_stream_status,init},{last_stats_report,{1501,381758,...}}],
> > trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> > reductions: 6919159795
> > ```
> >
> > Cause: This one looks similar to the one before. The trigger seems to be
> > that the source closes the connection for some reason, but the logs are
> > much more verbose this time.
> > Consequences: None. I think replicator just retries the task.
> >
> > Furthermore, this last logs show the username and passwords plain text on
> > the state key.
> >
> > Sorry for the wall of text.
> >
> > Regards
> >
> > --
> > [image: Cabify - Your private Driver] <http://www.cabify.com/>
> >
> > *Carlos Alonso*
> > Data Engineer
> > Madrid, Spain
> >
> > carlos.alonso@cabify.com
> >
> > Prueba gratis con este código
> > #CARLOSA6319 <https://cabify.com/i/carlosa6319>
> > [image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
> > <http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES
> >[image:
> > Linkedin] <https://www.linkedin.com/in/mrcalonso>
> >
> --
> [image: Cabify - Your private Driver] <http://www.cabify.com/>
>
> *Carlos Alonso*
> Data Engineer
> Madrid, Spain
>
> carlos.alonso@cabify.com
>
> Prueba gratis con este código
> #CARLOSA6319 <https://cabify.com/i/carlosa6319>
> [image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
> <http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
> Linkedin] <https://www.linkedin.com/in/mrcalonso>
>
> --
> Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
> destinatario, pudiendo contener información confidencial sometida a secreto
> profesional. No está permitida su reproducción o distribución sin la
> autorización expresa de Cabify. Si usted no es el destinatario final por
> favor elimínelo e infórmenos por esta vía.
>
> This message and any attached file are intended exclusively for the
> addressee, and it may be confidential. You are not allowed to copy or
> disclose it without Cabify's prior written authorization. If you are not
> the intended recipient please delete it from your system and notify us by
> e-mail.
>
--
[image: Cabify - Your private Driver] <http://www.cabify.com/>
*Carlos Alonso*
Data Engineer
Madrid, Spain
carlos.alonso@cabify.com
Prueba gratis con este código
#CARLOSA6319 <https://cabify.com/i/carlosa6319>
[image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
<http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
Linkedin] <https://www.linkedin.com/in/mrcalonso>
--
Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
destinatario, pudiendo contener información confidencial sometida a secreto
profesional. No está permitida su reproducción o distribución sin la
autorización expresa de Cabify. Si usted no es el destinatario final por
favor elimínelo e infórmenos por esta vía.
This message and any attached file are intended exclusively for the
addressee, and it may be confidential. You are not allowed to copy or
disclose it without Cabify's prior written authorization. If you are not
the intended recipient please delete it from your system and notify us by
e-mail.
Re: Understanding a couple replication errors
Posted by Carlos Alonso <ca...@cabify.com>.
Cool, thanks Joan!!
Is there any upgrade procedure available to get ready for it?
Regards
On Wed, Aug 2, 2017 at 11:47 PM Joan Touzet <wo...@apache.org> wrote:
> I haven't looked closely, but it could just be a side effect of
> #574. I recommend you upgrade to 2.1.0 (when it releases, very
> very very soon now) and see if the problem goes away first.
>
> -Joan
>
> ----- Original Message -----
> From: "Carlos Alonso" <ca...@cabify.com>
> To: user@couchdb.apache.org, "Joan Touzet" <wo...@apache.org>
> Sent: Wednesday, August 2, 2017 8:40:39 AM
> Subject: Re: Understanding a couple replication errors
>
>
> Hi Joan, many thanks for your reply
>
>
> Why would the timeout and closed connections happen? I see it quite often
> on the logs TBH but also, a slightly different one is a timeout against the
> cluster itself. What may this mean?
>
>
> [error] 2017-08-02T12:38:27.321697Z couchdb@couch-2 <0.7114.568> --------
> Replicator, request PUT to "http://admin:*****@
> 127.0.0.1:5984/my_db/35e78fe9d421677c77dea34522de2b99?new_edits=false "
> failed due to error {error,{conn_failed,{error,etimedout}}}
>
>
>
> Regards
>
>
> On Mon, Jul 31, 2017 at 5:06 PM Joan Touzet < wohali@apache.org > wrote:
>
>
> Hi Carlos,
>
> This issue (the {{nocatch,{mp_parser_died,noproc}}) is recently
> identified and patched!
>
> https://github.com/apache/couchdb/issues/574
>
> The other error just looks like a disconnection. You may want to
> wait (just a couple more days!) for 2.1.0 which has all sorts of
> new replication features and try again.
>
> -Joan
>
> ----- Original Message -----
> From: "Carlos Alonso" < carlos.alonso@cabify.com >
> To: "user" < user@couchdb.apache.org >
> Sent: Monday, July 31, 2017 9:58:09 AM
> Subject: Re: Understanding a couple replication errors
>
> On top of those, this one happens from time to time as well:
>
> [error] 2017-07-31T13:41:50.258423Z couchdb@couch-2 emulator --------
> Error
> in process <0.16620.2434> on node 'couchdb@couch-2' with exit value:
>
> {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},...
>
> [error] 2017-07-31T13:41:50.260569Z couchdb@couch-2 <0.32053.2433>
> --------
> Replicator, request PUT to "http://admin:*****@
> 127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false "
> failed due to error {error,
> {'EXIT',
> {{{nocatch,{mp_parser_died,noproc}},
> [{couch_att,'-foldl/4-fun-0-',3,
> [{file,"src/couch_att.erl"},{line,591}]},
> {couch_att,fold_streamed_data,4,
> [{file,"src/couch_att.erl"},{line,642}]},
> {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
> {couch_httpd_multipart,atts_to_mp,4,
> [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
> {gen_server,call,
> [<0.11924.2435>,
> {send_req,
> {{url,
> "
>
> http://admin:password@127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false
> ",
> "127.0.0.1",5984,"admin","password",
>
> "/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false",
> http,ipv4_address},
> [{"Accept","application/json"},
> {"Content-Length",32845},
> {"Content-Type",
> "multipart/related;
> boundary=\"46b37c9ba185ac67bf24ee40bcc38828\""},
> {"User-Agent","CouchDB-Replicator/2.0.0"}],
> put,
> {#Fun<couch_replicator_api_wrap.11.79455979>,
>
>
> {<<"{\"_id\":\"cba0aa16e347ac7d6b09732032828704\",\"_rev\":\"8-b90e6bd0b6281f303e587f4b9538a0af\",...\"_revisions\":{\"start\":8,\"ids\":[\"b90e6bd0b6281f303e587f4b9538a0af\",\"df66a380ea1258243bfd8a33506a18b4\",\"4cd932d1b71cebc4a55f91331d97f902\",\"a4dabd1dbf2b1de8987119a65cb44131\",\"82cc46c88ca768ff092de95c338bc0e9\",\"b4a7562ac8a831f46a194e97ea33e527\",\"d4fd4ddb998e4022e0335f609ed1ef50\",\"118cae6fc6864b3c48f788e78705c8cb\"]},\"_attachments\":{\"FF01-0020941.pdf\":{\"content_type\":\"application/pdf\",\"revpos\":8,\"digest\":\"md5-IT8Ms2HDG9EFLvfIxYTaVw==\",\"length\":21888,\"follows\":true},\"FF01-0020941.xml\":{\"content_type\":\"application/xml\",\"revpos\":8,\"digest\":\"md5-KJIXHDQbcJ3xeLaOUh11cw==\",\"length\":8526,\"follows\":true}}}">>,
> [{att,<<"FF01-0020941.pdf">>,<<"application/pdf">>,
> 21888,21888,
> <<33,63,12,179,97,195,27,209,5,46,247,200,197,132,
> 218,87>>,
> 8,
> {follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
> identity},
>
> {att,<<"FF01-0020941.xml">>,<<"application/xml">>,8526,
> 8526,
>
> <<40,146,23,28,52,27,112,157,241,120,182,142,82,29,
> 117,115>>,
> 8,
> {follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
> identity}],
> <<"46b37c9ba185ac67bf24ee40bcc38828">>,32845}},
> [{response_format,binary},
> {inactivity_timeout,30000},
> {socket_options,[{keepalive,true},{nodelay,false}]}],
> infinity}},
> infinity]}}}}
>
>
> Thanks!
>
>
>
> On Mon, Jul 31, 2017 at 11:52 AM Carlos Alonso < carlos.alonso@cabify.com
> >
> wrote:
>
> > Hi all!!
> >
> > I've been observing logs for a while and trying to understand why a few
> > errors happen while (pull) replication is happening and also trying to
> > understand the consequences.
> >
> > ** connection_closed*
> >
> > Log:
> > ```
> > [error] <0.15968.1946> -------- Replicator, request GET to
> "https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2210613426-g1AAAAOCeJyd0r1OwzAQB3CrRYIHYC8FCSZS2_kwEUM7svURiu1zVKqQSlUZYKFP0FeAV2DsBE_BCm-Suj4UKqRYIctZsv66n053OSHkaNoFcgpKzxdmBIrRQEt1lz1SygKdzx9AFsugMMvcZjuSqF5ZlrNpV5Hj8-t7-3eYZJyDzoD0qyZhTY8_lPBR6sRWNay0_pvTmJA8ScS-xhppPPJqo502qbSbwGlUUwNXrMVsiVe73WnPlXbx5DSQXBihW8xWF3NacWArWdnHgi8oXn7ifGHEeST_L_rPBMVXFN9RHORO5FowIeKm4m_KeywIfiD4heD6zIFpbCAL4xYrrIvtid8o_qxxvHGiiVKTQqM1zrad2hfx%22&timeout=10000"
> > failed due to error connection_closed
> > ```
> >
> > Cause: This one happens from time to time. It looks like the source is
> > closing connection, but why?
> > Consequences: None. I think replicator just retries the task.
> >
> > * *changes_reader_died*
> >
> > Log:
> > ```
> > [error] <0.15968.1946> -------- Replicator, request GET to
> "https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000"
> > failed due to error connection_closed
> >
> > [error] <0.13952.1946> -------- ChangesReader process died with reason:
> > {http_request_failed,"GET","
> > https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed
> > <https://admin:
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000%22,%7Berror,connection_closed>
> > }}
> >
> > [error] <0.13952.1946> -------- Replication
> > `363ed31e35361d024d22e0a990ca1ae5+continuous` (`https://admin:
> *****@source.ip:443/my_db/`
> > -> `http://admin:*****@ 127.0.0.1:5984/my_db/`
> <http://127.0.0.1:5984/my_db/>
> > < http://127.0.0.1:5984/my_db/ >) failed: changes_reader_died
> >
> > [error] <0.30404.6252> -------- Replicator, request GET to
> "https://admin:
> *****@source.ip:443/my_db/b666ee7ec8ed5b82ae035fe5fa6a4a2c?revs=true&open_revs=%5B%221-5a7ba8a8658bd0ab7b3656aa089aa183%22%5D&latest=true"
> > failed due to error {error,closing_on_request}
> >
> > [error] <0.352.0> -------- Error in replication
> > `363ed31e35361d024d22e0a990ca1ae5+continuous` (triggered by document
> > `28d973af08931f79fa423b54a8004b31`): changes_reader_died
> > Restarting replication in 5 seconds.
> >
> > [error] <0.13952.1946> -------- gen_server <0.13952.1946> terminated with
> > reason: changes_reader_died
> > last msg:
> > {'EXIT',<0.15968.1946>,{http_request_failed,"GET"," https://admin :
> >
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed}}}
> > state:
> >
> [{data,[{"State",{rep_state,{rep,{"363ed31e35361d024d22e0a990ca1ae5","+continuous"},{httpdb,"
> https://admin :
> >
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,nil,20},{httpdb,"
> http://admin :
> > *****@ 127.0.0.1:5984/my_db/
> >
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,nil,20},[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{http_connections,20},{retries,10},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}],{user_ctx,null,[],undefined},db,nil,<<"28d973af08931f79fa423b54a8004b31">>,<<"shards/c0000000-dfffffff/_replicator.1501167373">>},"
> https://admin :
> > *****@source.ip:443/my_db/","http://admin:*****@ 127.0.0.1:5984/my_db/
> > ",{httpdb," https://admin :
> >
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"
> http://admin :
> > *****@ 127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[],{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"source_last_seq">>,<<"36567936-g1...">>},{<<"replication_id_version">>,3},{<<"history">>,[{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"start_time">>,<<"Fri,
> > 28 Jul 2017 09:46:48 GMT">>},{<<"end_time">>,<<"Sun, 30 Jul 2017 02:28:59
> >
> GMT">>},{<<"start_last_seq">>,0},{<<"end_last_seq">>,<<"36567936-g1...">>},{<<"recorded_seq">>,<<"36567936-g1...">>},{<<"missing_checked">>,22991404},{<<"missing_found">>,22991404},{<<"docs_read">>,22991404},{<<"docs_written">>,22991404},{<<"doc_write_failures">>,0}]}]}]},{0,0},{46381,<<"36567936-g1...">>},{46388,<<"36574076-g1...">>},[{46389,<<"36575175-g1...">>},{46390,<<"36575938-g1...">>},{46391,<<"36576695-...">>},{46392,<<"36577391-g1...">>}],{46388,<<"36574076-g1...">>},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},"Fri,
> > 28 Jul 2017 09:46:48
> >
> GMT",<<"0">>,<<"0">>,nil,<0.16590.1946>,<0.16953.1946>,<0.15968.1946>,[<0.18469.1946>,<0.10009.1946>,<0.16617.1946>,<0.16155.1946>],[{doc_write_failures,0},{docs_read,22994904},{docs_written,22994904},{missing_checked,22994904},{missing_found,22994904}],<<"6f7aa01e879f5e1841cdb7100f40806b">>,nil,nil,nil,nil,<<"255587270-g1...">>,true,30000,db,nil}}]}]
> >
> > [error] <0.13952.1946> -------- CRASH REPORT Process (<0.13952.1946>)
> > with 2 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator,init,['Argument__1']}, ancestors:
> > [couch_replicator_job_sup,couch_replicator_sup,<0.349.0>], messages: [],
> > links: [<0.16590.1946>,<0.16953.1946>,<0.18469.1946>,<0.16617.1946>,...],
> > dictionary:
> >
> [{task_status_props,[{changes_pending,127404519},{checkpoint_interval,...},...]},...],
> > trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> > reductions: 26502159
> >
> > [error] <0.364.0> -------- Supervisor couch_replicator_job_sup had child
> > "363ed31e35361d024d22e0a990ca1ae5+continuous" started with
> > {gen_server,start_link,undefined} at <0.13952.1946> exit with reason
> > changes_reader_died in context child_terminated
> >
> > [error] <0.16617.1946> -------- gen_server <0.16617.1946> terminated with
> > reason: changes_reader_died
> > last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> > state: {state,<0.13952.1946>,<0.15004.1946>,20,{httpdb,"
> > https://admin:
> ******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@
> 127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.2293.6254>,<0.19041.6253>,<0.19785.6253>,<0.7470.6253>,<0.23117.6253>,<0.31026.6250>,<0.11251.6253>,<0.27821.6253>,<0.8477.6253>,<0.28980.6253>,<0.22848.6253>,<0.16549.6251>,<0.26208.6253>,<0.27365.6253>,<0.15830.6253>,<0.24330.6253>,<0.20562.6253>,<0.20869.6253>,<0.21799.6253>,<0.16311.6253>],nil,{{<0.15004.1946>,#Ref<0.0.6636.248998>},{<<"07597cb4d450c2b6ff674b675b9d5168">>,[{1,<<93,162,57,7,69,142,28,226,77,41,199,156,216,131,208,152>>}],[]}},nil,[{docs_read,199},{docs_written,25},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"b65rnw8pcm0jbt7jdpxvgmo1r\",\"_rev\":\"2-4d5283c4d125a0da73b95cc80b92abd6\
> > ",...}}
> >
> > [error] <0.16155.1946> -------- gen_server <0.16155.1946> terminated with
> > reason: changes_reader_died
> > last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> > state: {state,<0.13952.1946>,<0.21653.1946>,20,{httpdb,"
> > https://admin:
> ******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@
> 127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.27802.6253>,<0.27692.6253>,<0.10519.6253>,<0.5584.6253>,<0.9485.6253>,<0.2597.6253>,<0.12901.6253>,<0.21982.6253>,<0.20758.6253>,<0.12367.6253>,<0.30450.6253>,<0.15900.6253>,<0.17133.6253>,<0.22945.6253>,<0.8889.6253>,<0.20333.6253>,<0.12924.6253>,<0.32204.6252>,<0.32140.6253>,<0.5556.6253>],nil,{{<0.21653.1946>,#Ref<0.0.6636.251466>},{<<"9lcen85eu5ztrg8cs287sacoy">>,[{2,<<58,122,97,121,57,131,34,12,88,18,7,165,20,247,0,70>>}],[]}},nil,[{docs_read,436},{docs_written,57},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"1802v33hahs2ql9951vitjbih\",\"_rev\":\"5-138167bbafe87da23205b457c3918bd4\
> > ",...],...}}
> >
> > [error] <0.16617.1946> -------- CRASH REPORT Process (<0.16617.1946>)
> > with 15 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> > messages: [], links:
> > [<0.21799.6253>,<0.24330.6253>,<0.27365.6253>,<0.28980.6253>,...],
> > dictionary:
> > [{ibrowse_stream_status,init},{last_stats_report,{1501,381757,...}}],
> > trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> > reductions: 6921530028
> >
> > [error] <0.16155.1946> -------- CRASH REPORT Process (<0.16155.1946>)
> > with 15 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> > messages: [], links:
> > [<0.12924.6253>,<0.21982.6253>,<0.27692.6253>,<0.30450.6253>,...],
> > dictionary:
> > [{ibrowse_stream_status,init},{last_stats_report,{1501,381751,...}}],
> > trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> > reductions: 6914251593
> >
> > [error] <0.18469.1946> -------- gen_server <0.18469.1946> terminated with
> > reason: changes_reader_died
> > last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> > state: {state,<0.13952.1946>,<0.21935.1946>,20,{httpdb,"
> > https://admin:
> ******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@
> 127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.12414.6254>,<0.24987.6253>,<0.24685.6253>,<0.30475.6253>,<0.25412.6253>,<0.18441.6253>,<0.23627.6253>,<0.10529.6252>,<0.18425.6253>,<0.18994.6250>,<0.2214.6253>,<0.9544.6253>,<0.13738.6253>,<0.32462.6253>,<0.15946.6253>,<0.25304.6253>,<0.16690.6253>,<0.3468.6253>,<0.17659.6253>,<0.13537.6253>],nil,{{<0.21935.1946>,#Ref<0.0.6636.248486>},{<<"935cd70fbe4e02cb374180daa08a9fff">>,[{2,<<194,192,159,166,5,0,237,72,59,193,108,4,228,198,91,95>>}],[]}},nil,[{missing_checked,500},{missing_found,500}],nil,nil,{batch,[],0
> > }}
> >
> > [error] <0.10009.1946> -------- gen_server <0.10009.1946> terminated with
> > reason: changes_reader_died
> > last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> > state: {state,<0.13952.1946>,<0.22823.1946>,20,{httpdb,"
> > https://admin:
> ******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:
> ******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\
> > ",,...],...}}
> >
> > [error] <0.18469.1946> -------- CRASH REPORT Process (<0.18469.1946>)
> > with 15 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> > messages: [], links:
> > [<0.18441.6253>,<0.25304.6253>,<0.30475.6253>,<0.32462.6253>,...],
> > dictionary:
> > [{ibrowse_stream_status,init},{last_stats_report,{1501,381762,...}}],
> > trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> > reductions: 6918077761
> >
> > [error] <0.10009.1946> -------- CRASH REPORT Process (<0.10009.1946>)
> > with 15 neighbors exited with reason: changes_reader_died at
> > gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> > initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> > [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> > messages: [], links:
> > [<0.18053.6253>,<0.23938.6253>,<0.29957.6253>,<0.31079.6253>,...],
> > dictionary:
> > [{ibrowse_stream_status,init},{last_stats_report,{1501,381758,...}}],
> > trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> > reductions: 6919159795
> > ```
> >
> > Cause: This one looks similar to the one before. The trigger seems to be
> > that the source closes the connection for some reason, but the logs are
> > much more verbose this time.
> > Consequences: None. I think replicator just retries the task.
> >
> > Furthermore, this last logs show the username and passwords plain text on
> > the state key.
> >
> > Sorry for the wall of text.
> >
> > Regards
> >
> > --
> > [image: Cabify - Your private Driver] < http://www.cabify.com/ >
> >
> > *Carlos Alonso*
> > Data Engineer
> > Madrid, Spain
> >
> > carlos.alonso@cabify.com
> >
> > Prueba gratis con este código
> > #CARLOSA6319 < https://cabify.com/i/carlosa6319 >
> > [image: Facebook] < http://cbify.com/fb_ES >[image: Twitter]
> > < http://cbify.com/tw_ES >[image: Instagram] < http://cbify.com/in_ES
> >[image:
> > Linkedin] < https://www.linkedin.com/in/mrcalonso >
> >
> --
> [image: Cabify - Your private Driver] < http://www.cabify.com/ >
>
> *Carlos Alonso*
> Data Engineer
> Madrid, Spain
>
> carlos.alonso@cabify.com
>
> Prueba gratis con este código
> #CARLOSA6319 < https://cabify.com/i/carlosa6319 >
> [image: Facebook] < http://cbify.com/fb_ES >[image: Twitter]
> < http://cbify.com/tw_ES >[image: Instagram] < http://cbify.com/in_ES
> >[image:
> Linkedin] < https://www.linkedin.com/in/mrcalonso >
>
> --
> Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
> destinatario, pudiendo contener información confidencial sometida a secreto
> profesional. No está permitida su reproducción o distribución sin la
> autorización expresa de Cabify. Si usted no es el destinatario final por
> favor elimínelo e infórmenos por esta vía.
>
> This message and any attached file are intended exclusively for the
> addressee, and it may be confidential. You are not allowed to copy or
> disclose it without Cabify's prior written authorization. If you are not
> the intended recipient please delete it from your system and notify us by
> e-mail.
>
> --
>
>
> Cabify - Your private Driver
>
> Carlos Alonso
> Data Engineer
> Madrid, Spain
>
> carlos.alonso@cabify.com
>
>
>
> Prueba gratis con este código
> #CARLOSA6319
> FacebookTwitterInstagramLinkedin
> Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
> destinatario, pudiendo contener información confidencial sometida a secreto
> profesional. No está permitida su reproducción o distribución sin la
> autorización expresa de Cabify. Si usted no es el destinatario final por
> favor elimínelo e infórmenos por esta vía.
>
> This message and any attached file are intended exclusively for the
> addressee, and it may be confidential. You are not allowed to copy or
> disclose it without Cabify's prior written authorization. If you are not
> the intended recipient please delete it from your system and notify us by
> e-mail.
>
--
[image: Cabify - Your private Driver] <http://www.cabify.com/>
*Carlos Alonso*
Data Engineer
Madrid, Spain
carlos.alonso@cabify.com
Prueba gratis con este código
#CARLOSA6319 <https://cabify.com/i/carlosa6319>
[image: Facebook] <http://cbify.com/fb_ES>[image: Twitter]
<http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image:
Linkedin] <https://www.linkedin.com/in/mrcalonso>
--
Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
destinatario, pudiendo contener información confidencial sometida a secreto
profesional. No está permitida su reproducción o distribución sin la
autorización expresa de Cabify. Si usted no es el destinatario final por
favor elimínelo e infórmenos por esta vía.
This message and any attached file are intended exclusively for the
addressee, and it may be confidential. You are not allowed to copy or
disclose it without Cabify's prior written authorization. If you are not
the intended recipient please delete it from your system and notify us by
e-mail.
Re: Understanding a couple replication errors
Posted by Joan Touzet <wo...@apache.org>.
I haven't looked closely, but it could just be a side effect of
#574. I recommend you upgrade to 2.1.0 (when it releases, very
very very soon now) and see if the problem goes away first.
-Joan
----- Original Message -----
From: "Carlos Alonso" <ca...@cabify.com>
To: user@couchdb.apache.org, "Joan Touzet" <wo...@apache.org>
Sent: Wednesday, August 2, 2017 8:40:39 AM
Subject: Re: Understanding a couple replication errors
Hi Joan, many thanks for your reply
Why would the timeout and closed connections happen? I see it quite often on the logs TBH but also, a slightly different one is a timeout against the cluster itself. What may this mean?
[error] 2017-08-02T12:38:27.321697Z couchdb@couch-2 <0.7114.568> -------- Replicator, request PUT to "http://admin:*****@ 127.0.0.1:5984/my_db/35e78fe9d421677c77dea34522de2b99?new_edits=false " failed due to error {error,{conn_failed,{error,etimedout}}}
Regards
On Mon, Jul 31, 2017 at 5:06 PM Joan Touzet < wohali@apache.org > wrote:
Hi Carlos,
This issue (the {{nocatch,{mp_parser_died,noproc}}) is recently
identified and patched!
https://github.com/apache/couchdb/issues/574
The other error just looks like a disconnection. You may want to
wait (just a couple more days!) for 2.1.0 which has all sorts of
new replication features and try again.
-Joan
----- Original Message -----
From: "Carlos Alonso" < carlos.alonso@cabify.com >
To: "user" < user@couchdb.apache.org >
Sent: Monday, July 31, 2017 9:58:09 AM
Subject: Re: Understanding a couple replication errors
On top of those, this one happens from time to time as well:
[error] 2017-07-31T13:41:50.258423Z couchdb@couch-2 emulator -------- Error
in process <0.16620.2434> on node 'couchdb@couch-2' with exit value:
{{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},...
[error] 2017-07-31T13:41:50.260569Z couchdb@couch-2 <0.32053.2433> --------
Replicator, request PUT to "http://admin:*****@
127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false "
failed due to error {error,
{'EXIT',
{{{nocatch,{mp_parser_died,noproc}},
[{couch_att,'-foldl/4-fun-0-',3,
[{file,"src/couch_att.erl"},{line,591}]},
{couch_att,fold_streamed_data,4,
[{file,"src/couch_att.erl"},{line,642}]},
{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
{couch_httpd_multipart,atts_to_mp,4,
[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
{gen_server,call,
[<0.11924.2435>,
{send_req,
{{url,
"
http://admin:password@127.0.0.1:5984/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false
",
"127.0.0.1",5984,"admin","password",
"/my_db/cba0aa16e347ac7d6b09732032828704?new_edits=false",
http,ipv4_address},
[{"Accept","application/json"},
{"Content-Length",32845},
{"Content-Type",
"multipart/related;
boundary=\"46b37c9ba185ac67bf24ee40bcc38828\""},
{"User-Agent","CouchDB-Replicator/2.0.0"}],
put,
{#Fun<couch_replicator_api_wrap.11.79455979>,
{<<"{\"_id\":\"cba0aa16e347ac7d6b09732032828704\",\"_rev\":\"8-b90e6bd0b6281f303e587f4b9538a0af\",...\"_revisions\":{\"start\":8,\"ids\":[\"b90e6bd0b6281f303e587f4b9538a0af\",\"df66a380ea1258243bfd8a33506a18b4\",\"4cd932d1b71cebc4a55f91331d97f902\",\"a4dabd1dbf2b1de8987119a65cb44131\",\"82cc46c88ca768ff092de95c338bc0e9\",\"b4a7562ac8a831f46a194e97ea33e527\",\"d4fd4ddb998e4022e0335f609ed1ef50\",\"118cae6fc6864b3c48f788e78705c8cb\"]},\"_attachments\":{\"FF01-0020941.pdf\":{\"content_type\":\"application/pdf\",\"revpos\":8,\"digest\":\"md5-IT8Ms2HDG9EFLvfIxYTaVw==\",\"length\":21888,\"follows\":true},\"FF01-0020941.xml\":{\"content_type\":\"application/xml\",\"revpos\":8,\"digest\":\"md5-KJIXHDQbcJ3xeLaOUh11cw==\",\"length\":8526,\"follows\":true}}}">>,
[{att,<<"FF01-0020941.pdf">>,<<"application/pdf">>,
21888,21888,
<<33,63,12,179,97,195,27,209,5,46,247,200,197,132,
218,87>>,
8,
{follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
identity},
{att,<<"FF01-0020941.xml">>,<<"application/xml">>,8526,
8526,
<<40,146,23,28,52,27,112,157,241,120,182,142,82,29,
117,115>>,
8,
{follows,<0.31188.2433>,#Ref<0.0.12124.197441>},
identity}],
<<"46b37c9ba185ac67bf24ee40bcc38828">>,32845}},
[{response_format,binary},
{inactivity_timeout,30000},
{socket_options,[{keepalive,true},{nodelay,false}]}],
infinity}},
infinity]}}}}
Thanks!
On Mon, Jul 31, 2017 at 11:52 AM Carlos Alonso < carlos.alonso@cabify.com >
wrote:
> Hi all!!
>
> I've been observing logs for a while and trying to understand why a few
> errors happen while (pull) replication is happening and also trying to
> understand the consequences.
>
> ** connection_closed*
>
> Log:
> ```
> [error] <0.15968.1946> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2210613426-g1AAAAOCeJyd0r1OwzAQB3CrRYIHYC8FCSZS2_kwEUM7svURiu1zVKqQSlUZYKFP0FeAV2DsBE_BCm-Suj4UKqRYIctZsv66n053OSHkaNoFcgpKzxdmBIrRQEt1lz1SygKdzx9AFsugMMvcZjuSqF5ZlrNpV5Hj8-t7-3eYZJyDzoD0qyZhTY8_lPBR6sRWNay0_pvTmJA8ScS-xhppPPJqo502qbSbwGlUUwNXrMVsiVe73WnPlXbx5DSQXBihW8xWF3NacWArWdnHgi8oXn7ifGHEeST_L_rPBMVXFN9RHORO5FowIeKm4m_KeywIfiD4heD6zIFpbCAL4xYrrIvtid8o_qxxvHGiiVKTQqM1zrad2hfx%22&timeout=10000"
> failed due to error connection_closed
> ```
>
> Cause: This one happens from time to time. It looks like the source is
> closing connection, but why?
> Consequences: None. I think replicator just retries the task.
>
> * *changes_reader_died*
>
> Log:
> ```
> [error] <0.15968.1946> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000"
> failed due to error connection_closed
>
> [error] <0.13952.1946> -------- ChangesReader process died with reason:
> {http_request_failed,"GET","
> https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed
> <https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000%22,%7Berror,connection_closed>
> }}
>
> [error] <0.13952.1946> -------- Replication
> `363ed31e35361d024d22e0a990ca1ae5+continuous` (`https://admin:*****@source.ip:443/my_db/`
> -> `http://admin:*****@ 127.0.0.1:5984/my_db/`
> < http://127.0.0.1:5984/my_db/ >) failed: changes_reader_died
>
> [error] <0.30404.6252> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/b666ee7ec8ed5b82ae035fe5fa6a4a2c?revs=true&open_revs=%5B%221-5a7ba8a8658bd0ab7b3656aa089aa183%22%5D&latest=true"
> failed due to error {error,closing_on_request}
>
> [error] <0.352.0> -------- Error in replication
> `363ed31e35361d024d22e0a990ca1ae5+continuous` (triggered by document
> `28d973af08931f79fa423b54a8004b31`): changes_reader_died
> Restarting replication in 5 seconds.
>
> [error] <0.13952.1946> -------- gen_server <0.13952.1946> terminated with
> reason: changes_reader_died
> last msg:
> {'EXIT',<0.15968.1946>,{http_request_failed,"GET"," https://admin :
> *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed}}}
> state:
> [{data,[{"State",{rep_state,{rep,{"363ed31e35361d024d22e0a990ca1ae5","+continuous"},{httpdb," https://admin :
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,nil,20},{httpdb," http://admin :
> *****@ 127.0.0.1:5984/my_db/
> ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,nil,20},[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{http_connections,20},{retries,10},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}],{user_ctx,null,[],undefined},db,nil,<<"28d973af08931f79fa423b54a8004b31">>,<<"shards/c0000000-dfffffff/_replicator.1501167373">>}," https://admin :
> *****@source.ip:443/my_db/","http://admin:*****@ 127.0.0.1:5984/my_db/
> ",{httpdb," https://admin :
> *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb," http://admin :
> *****@ 127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[],{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"source_last_seq">>,<<"36567936-g1...">>},{<<"replication_id_version">>,3},{<<"history">>,[{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"start_time">>,<<"Fri,
> 28 Jul 2017 09:46:48 GMT">>},{<<"end_time">>,<<"Sun, 30 Jul 2017 02:28:59
> GMT">>},{<<"start_last_seq">>,0},{<<"end_last_seq">>,<<"36567936-g1...">>},{<<"recorded_seq">>,<<"36567936-g1...">>},{<<"missing_checked">>,22991404},{<<"missing_found">>,22991404},{<<"docs_read">>,22991404},{<<"docs_written">>,22991404},{<<"doc_write_failures">>,0}]}]}]},{0,0},{46381,<<"36567936-g1...">>},{46388,<<"36574076-g1...">>},[{46389,<<"36575175-g1...">>},{46390,<<"36575938-g1...">>},{46391,<<"36576695-...">>},{46392,<<"36577391-g1...">>}],{46388,<<"36574076-g1...">>},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},"Fri,
> 28 Jul 2017 09:46:48
> GMT",<<"0">>,<<"0">>,nil,<0.16590.1946>,<0.16953.1946>,<0.15968.1946>,[<0.18469.1946>,<0.10009.1946>,<0.16617.1946>,<0.16155.1946>],[{doc_write_failures,0},{docs_read,22994904},{docs_written,22994904},{missing_checked,22994904},{missing_found,22994904}],<<"6f7aa01e879f5e1841cdb7100f40806b">>,nil,nil,nil,nil,<<"255587270-g1...">>,true,30000,db,nil}}]}]
>
> [error] <0.13952.1946> -------- CRASH REPORT Process (<0.13952.1946>)
> with 2 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator,init,['Argument__1']}, ancestors:
> [couch_replicator_job_sup,couch_replicator_sup,<0.349.0>], messages: [],
> links: [<0.16590.1946>,<0.16953.1946>,<0.18469.1946>,<0.16617.1946>,...],
> dictionary:
> [{task_status_props,[{changes_pending,127404519},{checkpoint_interval,...},...]},...],
> trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> reductions: 26502159
>
> [error] <0.364.0> -------- Supervisor couch_replicator_job_sup had child
> "363ed31e35361d024d22e0a990ca1ae5+continuous" started with
> {gen_server,start_link,undefined} at <0.13952.1946> exit with reason
> changes_reader_died in context child_terminated
>
> [error] <0.16617.1946> -------- gen_server <0.16617.1946> terminated with
> reason: changes_reader_died
> last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> state: {state,<0.13952.1946>,<0.15004.1946>,20,{httpdb,"
> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ 127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.2293.6254>,<0.19041.6253>,<0.19785.6253>,<0.7470.6253>,<0.23117.6253>,<0.31026.6250>,<0.11251.6253>,<0.27821.6253>,<0.8477.6253>,<0.28980.6253>,<0.22848.6253>,<0.16549.6251>,<0.26208.6253>,<0.27365.6253>,<0.15830.6253>,<0.24330.6253>,<0.20562.6253>,<0.20869.6253>,<0.21799.6253>,<0.16311.6253>],nil,{{<0.15004.1946>,#Ref<0.0.6636.248998>},{<<"07597cb4d450c2b6ff674b675b9d5168">>,[{1,<<93,162,57,7,69,142,28,226,77,41,199,156,216,131,208,152>>}],[]}},nil,[{docs_read,199},{docs_written,25},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"b65rnw8pcm0jbt7jdpxvgmo1r\",\"_rev\":\"2-4d5283c4d125a0da73b95cc80b92abd6\
> ",...}}
>
> [error] <0.16155.1946> -------- gen_server <0.16155.1946> terminated with
> reason: changes_reader_died
> last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> state: {state,<0.13952.1946>,<0.21653.1946>,20,{httpdb,"
> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ 127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.27802.6253>,<0.27692.6253>,<0.10519.6253>,<0.5584.6253>,<0.9485.6253>,<0.2597.6253>,<0.12901.6253>,<0.21982.6253>,<0.20758.6253>,<0.12367.6253>,<0.30450.6253>,<0.15900.6253>,<0.17133.6253>,<0.22945.6253>,<0.8889.6253>,<0.20333.6253>,<0.12924.6253>,<0.32204.6252>,<0.32140.6253>,<0.5556.6253>],nil,{{<0.21653.1946>,#Ref<0.0.6636.251466>},{<<"9lcen85eu5ztrg8cs287sacoy">>,[{2,<<58,122,97,121,57,131,34,12,88,18,7,165,20,247,0,70>>}],[]}},nil,[{docs_read,436},{docs_written,57},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"1802v33hahs2ql9951vitjbih\",\"_rev\":\"5-138167bbafe87da23205b457c3918bd4\
> ",...],...}}
>
> [error] <0.16617.1946> -------- CRASH REPORT Process (<0.16617.1946>)
> with 15 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> messages: [], links:
> [<0.21799.6253>,<0.24330.6253>,<0.27365.6253>,<0.28980.6253>,...],
> dictionary:
> [{ibrowse_stream_status,init},{last_stats_report,{1501,381757,...}}],
> trap_exit: true, status: running, heap_size: 4185, stack_size: 27,
> reductions: 6921530028
>
> [error] <0.16155.1946> -------- CRASH REPORT Process (<0.16155.1946>)
> with 15 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> messages: [], links:
> [<0.12924.6253>,<0.21982.6253>,<0.27692.6253>,<0.30450.6253>,...],
> dictionary:
> [{ibrowse_stream_status,init},{last_stats_report,{1501,381751,...}}],
> trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> reductions: 6914251593
>
> [error] <0.18469.1946> -------- gen_server <0.18469.1946> terminated with
> reason: changes_reader_died
> last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> state: {state,<0.13952.1946>,<0.21935.1946>,20,{httpdb,"
> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@ 127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.12414.6254>,<0.24987.6253>,<0.24685.6253>,<0.30475.6253>,<0.25412.6253>,<0.18441.6253>,<0.23627.6253>,<0.10529.6252>,<0.18425.6253>,<0.18994.6250>,<0.2214.6253>,<0.9544.6253>,<0.13738.6253>,<0.32462.6253>,<0.15946.6253>,<0.25304.6253>,<0.16690.6253>,<0.3468.6253>,<0.17659.6253>,<0.13537.6253>],nil,{{<0.21935.1946>,#Ref<0.0.6636.248486>},{<<"935cd70fbe4e02cb374180daa08a9fff">>,[{2,<<194,192,159,166,5,0,237,72,59,193,108,4,228,198,91,95>>}],[]}},nil,[{missing_checked,500},{missing_found,500}],nil,nil,{batch,[],0
> }}
>
> [error] <0.10009.1946> -------- gen_server <0.10009.1946> terminated with
> reason: changes_reader_died
> last msg: {'EXIT',<0.13952.1946>,changes_reader_died}
> state: {state,<0.13952.1946>,<0.22823.1946>,20,{httpdb,"
> https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\
> ",,...],...}}
>
> [error] <0.18469.1946> -------- CRASH REPORT Process (<0.18469.1946>)
> with 15 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> messages: [], links:
> [<0.18441.6253>,<0.25304.6253>,<0.30475.6253>,<0.32462.6253>,...],
> dictionary:
> [{ibrowse_stream_status,init},{last_stats_report,{1501,381762,...}}],
> trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> reductions: 6918077761
>
> [error] <0.10009.1946> -------- CRASH REPORT Process (<0.10009.1946>)
> with 15 neighbors exited with reason: changes_reader_died at
> gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237);
> initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...],
> messages: [], links:
> [<0.18053.6253>,<0.23938.6253>,<0.29957.6253>,<0.31079.6253>,...],
> dictionary:
> [{ibrowse_stream_status,init},{last_stats_report,{1501,381758,...}}],
> trap_exit: true, status: running, heap_size: 6772, stack_size: 27,
> reductions: 6919159795
> ```
>
> Cause: This one looks similar to the one before. The trigger seems to be
> that the source closes the connection for some reason, but the logs are
> much more verbose this time.
> Consequences: None. I think replicator just retries the task.
>
> Furthermore, this last logs show the username and passwords plain text on
> the state key.
>
> Sorry for the wall of text.
>
> Regards
>
> --
> [image: Cabify - Your private Driver] < http://www.cabify.com/ >
>
> *Carlos Alonso*
> Data Engineer
> Madrid, Spain
>
> carlos.alonso@cabify.com
>
> Prueba gratis con este código
> #CARLOSA6319 < https://cabify.com/i/carlosa6319 >
> [image: Facebook] < http://cbify.com/fb_ES >[image: Twitter]
> < http://cbify.com/tw_ES >[image: Instagram] < http://cbify.com/in_ES >[image:
> Linkedin] < https://www.linkedin.com/in/mrcalonso >
>
--
[image: Cabify - Your private Driver] < http://www.cabify.com/ >
*Carlos Alonso*
Data Engineer
Madrid, Spain
carlos.alonso@cabify.com
Prueba gratis con este código
#CARLOSA6319 < https://cabify.com/i/carlosa6319 >
[image: Facebook] < http://cbify.com/fb_ES >[image: Twitter]
< http://cbify.com/tw_ES >[image: Instagram] < http://cbify.com/in_ES >[image:
Linkedin] < https://www.linkedin.com/in/mrcalonso >
--
Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su
destinatario, pudiendo contener información confidencial sometida a secreto
profesional. No está permitida su reproducción o distribución sin la
autorización expresa de Cabify. Si usted no es el destinatario final por
favor elimínelo e infórmenos por esta vía.
This message and any attached file are intended exclusively for the
addressee, and it may be confidential. You are not allowed to copy or
disclose it without Cabify's prior written authorization. If you are not
the intended recipient please delete it from your system and notify us by
e-mail.
--
Cabify - Your private Driver
Carlos Alonso
Data Engineer
Madrid, Spain
carlos.alonso@cabify.com
Prueba gratis con este código
#CARLOSA6319
FacebookTwitterInstagramLinkedin
Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su destinatario, pudiendo contener información confidencial sometida a secreto profesional. No está permitida su reproducción o distribución sin la autorización expresa de Cabify. Si usted no es el destinatario final por favor elimínelo e infórmenos por esta vía.
This message and any attached file are intended exclusively for the addressee, and it may be confidential. You are not allowed to copy or disclose it without Cabify's prior written authorization. If you are not the intended recipient please delete it from your system and notify us by e-mail.