You are viewing a plain text version of this content. The canonical link for it is here.
Posted to replication@couchdb.apache.org by Jens Alfke <je...@couchbase.com> on 2014/04/22 19:23:37 UTC
CouchDB 1.5 throwing exceptions pulling attachments from Couchbase Sync Gateway
A developer is reporting that a CouchDB pull replication, with source the Couchbase Sync Gateway, is failing with Erlang exceptions. The cause appears to be a MIME multipart response from a GET of a document with attachments — he says “the response of SG GET /nam/image-97m333b8iah?revs=true&open_revs=%5B%229-9f060576f801f3c9cea5c90013c6d182%22%5D&latest=true properly returns a multipart document, but it seems CouchDB 1.5 is unable to parse the data.”
I don’t think this is the known-and-recently-fixed issue with CouchDB and chunked encoding. Replicating attachments to CouchDB has been working for a while, and the developer reports "the documents only containing 2 images did replicate from SG to CouchDB 1.5.” It may be that the exact length or contents of this document are triggering some kind of edge case in CouchDB.
Anyway, I can’t make head or tail of the Erlang crash logs; could someone skilled in the art provide some insight into what happened to CouchDB? Thanks!
—Jens
> Using CouchDB 1.3 the pull replication fails completely. Erlang throws some errors and completely restarts.
>
> Using CouchDB 1.5 the pull replication does not fail, but is never completed. It seems CouchDB is unable to parse the multipart response.
> CouchDB 1.5 logs:
>
> [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17159.1>] Replication crashing because GET http://192.168.1.2:4985/nam/image-97m333b8iah?revs=true&open_revs=%5B%229-9f060576f801f3c9cea5c90013c6d182%22%5D&latest=true failed
> [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17142.1>] ** Generic server <0.17142.1> terminating
> ** Last message in was {'EXIT',<0.17159.1>,kaboom}
> ** When Server state == {state,<0.17123.1>,<0.17143.1>,20,
> {httpdb,"http://192.168.1.2:4985/nam/",nil,
> [{"Accept","application/json"},
> {"User-Agent","CouchDB/1.5.0"}],
> 30000,
> [{socket_options,
> [{keepalive,true},{nodelay,false}]}],
> 10,250,<0.17124.1>,20},
> {db,<0.17131.1>,<0.17132.1>,nil,
> <<"1398170695581111">>,<0.17133.1>,
> <0.17129.1>,<0.17135.1>,
> {db_header,6,9,0,
> {2230287,{3,6,2227482},1191},
> {2228708,9,508},
> {2232424,[],807},
> 0,nil,nil,1000},
> 9,
> {btree,<0.17129.1>,
> {2230287,{3,6,2227482},1191},
> #Fun<couch_db_updater.10.75275056>,
> #Fun<couch_db_updater.11.75275056>,
> #Fun<couch_btree.5.71351651>,
> #Fun<couch_db_updater.12.75275056>,snappy},
> {btree,<0.17129.1>,
> {2228708,9,508},
> #Fun<couch_db_updater.13.75275056>,
> #Fun<couch_db_updater.14.75275056>,
> #Fun<couch_btree.5.71351651>,
> #Fun<couch_db_updater.15.75275056>,snappy},
> {btree,<0.17129.1>,
> {2232424,[],807},
> #Fun<couch_btree.3.71351651>,
> #Fun<couch_btree.4.71351651>,
> #Fun<couch_btree.5.71351651>,nil,snappy},
> 9,<<"snagtracker-140417-tenpost">>,
> "/usr/local/var/lib/couchdb/snagtracker-140417-tenpost.couch",
> [],[],nil,
> {user_ctx,null,[],undefined},
> nil,1000,
> [before_header,after_header,on_file_open],
> [{user_ctx,{user_ctx,null,[],undefined}}],
> snappy,nil,nil},
> [<0.17159.1>],
> nil,nil,
> {<0.17143.1>,#Ref<0.0.2.127076>},
> {rep_stats,1,1,0,0,0},
> nil,<0.17144.1>,
> {batch,[],0}}
> ** Reason for termination ==
> ** {process_died,<0.17159.1>,kaboom}
>
> [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17142.1>] {error_report,<0.17006.1>,
> {<0.17142.1>,crash_report,
> [[{initial_call,
> {couch_replicator_worker,init,['Argument__1']}},
> {pid,<0.17142.1>},
> {registered_name,[]},
> {error_info,
> {exit,
> {process_died,<0.17159.1>,kaboom},
> [{gen_server,terminate,6,
> [{file,"gen_server.erl"},{line,744}]},
> {proc_lib,init_p_do_apply,3,
> [{file,"proc_lib.erl"},{line,239}]}]}},
> {ancestors,
> [<0.17123.1>,couch_replicator_job_sup,
> couch_primary_services,couch_server_sup,
> <0.17007.1>]},
> {messages,[{'EXIT',<0.17144.1>,normal}]},
> {links,[<0.17143.1>,<0.17123.1>]},
> {dictionary,
> [{last_stats_report,{1398,170695,606206}}]},
> {trap_exit,true},
> {status,running},
> {heap_size,1598},
> {stack_size,27},
> {reductions,305}],
> [{neighbour,
> [{pid,<0.17143.1>},
> {registered_name,[]},
> {initial_call,{erlang,apply,2}},
> {current_function,{gen,do_call,4}},
> {ancestors,[]},
> {messages,[]},
> {links,[<0.17142.1>]},
> {dictionary,[]},
> {trap_exit,false},
> {status,waiting},
> {heap_size,1598},
> {stack_size,22},
> {reductions,249}]}]]}}
Re: CouchDB 1.5 throwing exceptions pulling attachments from Couchbase Sync Gateway
Posted by Jan Lehnardt <ja...@apache.org>.
On 24 Apr 2014, at 14:27 , Alexander Shorin <kx...@gmail.com> wrote:
> On Thu, Apr 24, 2014 at 2:50 AM, Jens Alfke <je...@couchbase.com> wrote:
>> I got more information from the developer, specifically some more logs. Everything’s in a Github issue:
>> https://github.com/couchbase/sync_gateway/issues/319
>> The CouchDB 1.5 logs are here:
>> https://gist.github.com/ssured/11230023
>> and CouchDB 1.3 logs, where it takes down the whole server:
>> https://gist.github.com/ssured/11229955
>
> Thanks Jens, but logs doesn't really answers on question about "what's
> exactly wrong". Having actual communication dump would be more helpful
> to diag the roots of the issue.
>
> I suggest to move this discussion to the github issue to keep all the
> things in single place.
Good call Alex. It’d be nice to get a pointer sent to this list when the issue closes.
Best
Jan
--
Re: CouchDB 1.5 throwing exceptions pulling attachments from
Couchbase Sync Gateway
Posted by Alexander Shorin <kx...@gmail.com>.
On Thu, Apr 24, 2014 at 2:50 AM, Jens Alfke <je...@couchbase.com> wrote:
> I got more information from the developer, specifically some more logs. Everything’s in a Github issue:
> https://github.com/couchbase/sync_gateway/issues/319
> The CouchDB 1.5 logs are here:
> https://gist.github.com/ssured/11230023
> and CouchDB 1.3 logs, where it takes down the whole server:
> https://gist.github.com/ssured/11229955
Thanks Jens, but logs doesn't really answers on question about "what's
exactly wrong". Having actual communication dump would be more helpful
to diag the roots of the issue.
I suggest to move this discussion to the github issue to keep all the
things in single place.
--
,,,^..^,,,
Re: CouchDB 1.5 throwing exceptions pulling attachments from Couchbase Sync Gateway
Posted by Jens Alfke <je...@couchbase.com>.
I got more information from the developer, specifically some more logs. Everything’s in a Github issue:
https://github.com/couchbase/sync_gateway/issues/319
The CouchDB 1.5 logs are here:
https://gist.github.com/ssured/11230023
and CouchDB 1.3 logs, where it takes down the whole server:
https://gist.github.com/ssured/11229955
—Jens
Re: CouchDB 1.5 throwing exceptions pulling attachments from Couchbase Sync Gateway
Posted by Adam Kocoloski <ko...@apache.org>.
Jens, there should be lines earlier in the logfile of the form:
> "Retrying GET to ~s in ~p seconds due to error ~p"
It may be useful to see what the reported error is there.
Adam
On Apr 22, 2014, at 2:16 PM, Alexander Shorin <kx...@gmail.com> wrote:
> On Tue, Apr 22, 2014 at 10:09 PM, Jens Alfke <je...@couchbase.com> wrote:
>> On Apr 22, 2014, at 10:28 AM, Alexander Shorin <kx...@gmail.com> wrote:
>>
>>> Any chance to get these files / see actual request to reproduce the issue?
>>
>> I’m asking the dev for more info. I’ll let you know.
>
> Thanks! That would be very helpful.
>
>>> Does it happens for couchdb-to-couchdb replication?
>>
>> He didn’t say. It sounds like he’s primarily syncing between Couchbase Lite and Sync Gateway, but in this one case he wanted to copy his database to CouchDB. I’m betting it wouldn’t happen for CouchDB-to-CouchDB; this is probably some slight disagreement over multipart syntax, as has happened before.
>
> Just curious about attempt to compare responses of CouchDB and
> Couchbase Lite for the same request on the same data. The diff of
> responses should contains the answer.
>
>
> --
> ,,,^..^,,,
Re: CouchDB 1.5 throwing exceptions pulling attachments from
Couchbase Sync Gateway
Posted by Alexander Shorin <kx...@gmail.com>.
On Tue, Apr 22, 2014 at 10:09 PM, Jens Alfke <je...@couchbase.com> wrote:
> On Apr 22, 2014, at 10:28 AM, Alexander Shorin <kx...@gmail.com> wrote:
>
>> Any chance to get these files / see actual request to reproduce the issue?
>
> I’m asking the dev for more info. I’ll let you know.
Thanks! That would be very helpful.
>> Does it happens for couchdb-to-couchdb replication?
>
> He didn’t say. It sounds like he’s primarily syncing between Couchbase Lite and Sync Gateway, but in this one case he wanted to copy his database to CouchDB. I’m betting it wouldn’t happen for CouchDB-to-CouchDB; this is probably some slight disagreement over multipart syntax, as has happened before.
Just curious about attempt to compare responses of CouchDB and
Couchbase Lite for the same request on the same data. The diff of
responses should contains the answer.
--
,,,^..^,,,
Re: CouchDB 1.5 throwing exceptions pulling attachments from Couchbase Sync Gateway
Posted by Jens Alfke <je...@couchbase.com>.
On Apr 22, 2014, at 10:28 AM, Alexander Shorin <kx...@gmail.com> wrote:
> Any chance to get these files / see actual request to reproduce the issue?
I’m asking the dev for more info. I’ll let you know.
> Does it happens for couchdb-to-couchdb replication?
He didn’t say. It sounds like he’s primarily syncing between Couchbase Lite and Sync Gateway, but in this one case he wanted to copy his database to CouchDB. I’m betting it wouldn’t happen for CouchDB-to-CouchDB; this is probably some slight disagreement over multipart syntax, as has happened before.
—Jens
Re: CouchDB 1.5 throwing exceptions pulling attachments from
Couchbase Sync Gateway
Posted by Alexander Shorin <kx...@gmail.com>.
Hi Jens,
Any chance to get these files / see actual request to reproduce the issue?
Does it happens for couchdb-to-couchdb replication?
,,,^..^,,,
On 22 Apr 2014 21:24, "Jens Alfke" <je...@couchbase.com> wrote:
> A developer is reporting that a CouchDB pull replication, with source the
> Couchbase Sync Gateway, is failing with Erlang exceptions. The cause
> appears to be a MIME multipart response from a GET of a document with
> attachments — he says “the response of SG GET
> /nam/image-97m333b8iah?revs=true&open_revs=%5B%229-9f060576f801f3c9cea5c90013c6d182%22%5D&latest=true
> properly returns a multipart document, but it seems CouchDB 1.5 is unable
> to parse the data.”
>
> I don’t think this is the known-and-recently-fixed issue with CouchDB and
> chunked encoding. Replicating attachments to CouchDB has been working for a
> while, and the developer reports "the documents only containing 2 images
> did replicate from SG to CouchDB 1.5.” It may be that the exact length or
> contents of this document are triggering some kind of edge case in CouchDB.
>
> Anyway, I can’t make head or tail of the Erlang crash logs; could someone
> skilled in the art provide some insight into what happened to CouchDB?
> Thanks!
>
> —Jens
>
> > Using CouchDB 1.3 the pull replication fails completely. Erlang throws
> some errors and completely restarts.
> >
> > Using CouchDB 1.5 the pull replication does not fail, but is never
> completed. It seems CouchDB is unable to parse the multipart response.
> > CouchDB 1.5 logs:
> >
> > [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17159.1>] Replication
> crashing because GET
> http://192.168.1.2:4985/nam/image-97m333b8iah?revs=true&open_revs=%5B%229-9f060576f801f3c9cea5c90013c6d182%22%5D&latest=truefailed
> > [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17142.1>] ** Generic server
> <0.17142.1> terminating
> > ** Last message in was {'EXIT',<0.17159.1>,kaboom}
> > ** When Server state == {state,<0.17123.1>,<0.17143.1>,20,
> > {httpdb,"http://192.168.1.2:4985/nam/",nil,
> > [{"Accept","application/json"},
> > {"User-Agent","CouchDB/1.5.0"}],
> > 30000,
> > [{socket_options,
> >
> [{keepalive,true},{nodelay,false}]}],
> > 10,250,<0.17124.1>,20},
> > {db,<0.17131.1>,<0.17132.1>,nil,
> > <<"1398170695581111">>,<0.17133.1>,
> > <0.17129.1>,<0.17135.1>,
> > {db_header,6,9,0,
> > {2230287,{3,6,2227482},1191},
> > {2228708,9,508},
> > {2232424,[],807},
> > 0,nil,nil,1000},
> > 9,
> > {btree,<0.17129.1>,
> > {2230287,{3,6,2227482},1191},
> > #Fun<couch_db_updater.10.75275056>,
> > #Fun<couch_db_updater.11.75275056>,
> > #Fun<couch_btree.5.71351651>,
> >
> #Fun<couch_db_updater.12.75275056>,snappy},
> > {btree,<0.17129.1>,
> > {2228708,9,508},
> > #Fun<couch_db_updater.13.75275056>,
> > #Fun<couch_db_updater.14.75275056>,
> > #Fun<couch_btree.5.71351651>,
> >
> #Fun<couch_db_updater.15.75275056>,snappy},
> > {btree,<0.17129.1>,
> > {2232424,[],807},
> > #Fun<couch_btree.3.71351651>,
> > #Fun<couch_btree.4.71351651>,
> >
> #Fun<couch_btree.5.71351651>,nil,snappy},
> > 9,<<"snagtracker-140417-tenpost">>,
> >
> "/usr/local/var/lib/couchdb/snagtracker-140417-tenpost.couch",
> > [],[],nil,
> > {user_ctx,null,[],undefined},
> > nil,1000,
> >
> [before_header,after_header,on_file_open],
> >
> [{user_ctx,{user_ctx,null,[],undefined}}],
> > snappy,nil,nil},
> > [<0.17159.1>],
> > nil,nil,
> > {<0.17143.1>,#Ref<0.0.2.127076>},
> > {rep_stats,1,1,0,0,0},
> > nil,<0.17144.1>,
> > {batch,[],0}}
> > ** Reason for termination ==
> > ** {process_died,<0.17159.1>,kaboom}
> >
> > [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17142.1>]
> {error_report,<0.17006.1>,
> > {<0.17142.1>,crash_report,
> > [[{initial_call,
> >
> {couch_replicator_worker,init,['Argument__1']}},
> > {pid,<0.17142.1>},
> > {registered_name,[]},
> > {error_info,
> > {exit,
> > {process_died,<0.17159.1>,kaboom},
> > [{gen_server,terminate,6,
> > [{file,"gen_server.erl"},{line,744}]},
> > {proc_lib,init_p_do_apply,3,
> > [{file,"proc_lib.erl"},{line,239}]}]}},
> > {ancestors,
> > [<0.17123.1>,couch_replicator_job_sup,
> > couch_primary_services,couch_server_sup,
> > <0.17007.1>]},
> > {messages,[{'EXIT',<0.17144.1>,normal}]},
> > {links,[<0.17143.1>,<0.17123.1>]},
> > {dictionary,
> > [{last_stats_report,{1398,170695,606206}}]},
> > {trap_exit,true},
> > {status,running},
> > {heap_size,1598},
> > {stack_size,27},
> > {reductions,305}],
> > [{neighbour,
> > [{pid,<0.17143.1>},
> > {registered_name,[]},
> > {initial_call,{erlang,apply,2}},
> > {current_function,{gen,do_call,4}},
> > {ancestors,[]},
> > {messages,[]},
> > {links,[<0.17142.1>]},
> > {dictionary,[]},
> > {trap_exit,false},
> > {status,waiting},
> > {heap_size,1598},
> > {stack_size,22},
> > {reductions,249}]}]]}}
>
Re: CouchDB 1.5 throwing exceptions pulling attachments from
Couchbase Sync Gateway
Posted by Alexander Shorin <kx...@gmail.com>.
On Tue, Apr 22, 2014 at 10:08 PM, Calvin Metcalf
<ca...@gmail.com> wrote:
>> ** When Server state == {state,<0.17123.1>,<0.17143.1>,20,
>> {httpdb,"http://192.168.1.2:4985/nam/",nil,
>> [{"Accept","application/json"},
>> {"User-Agent","CouchDB/1.5.0"}],
>
> is this not what I think it is?
Exactly.
--
,,,^..^,,,
Re: CouchDB 1.5 throwing exceptions pulling attachments from
Couchbase Sync Gateway
Posted by Calvin Metcalf <ca...@gmail.com>.
> ** When Server state == {state,<0.17123.1>,<0.17143.1>,20,
> {httpdb,"http://192.168.1.2:4985/nam/",nil,
> [{"Accept","application/json"},
> {"User-Agent","CouchDB/1.5.0"}],
is this not what I think it is?
On Tue, Apr 22, 2014 at 1:52 PM, Alexander Shorin <kx...@gmail.com> wrote:
> On Tue, Apr 22, 2014 at 9:30 PM, Calvin Metcalf
> <ca...@gmail.com> wrote:
> > Looks like couch
> > is sending {"Accept","application/json"} in this request as well.
>
> This request is unrelated to fetching document revisions and his
> attachments, it's about fetching database information, which runs
> periodically during replication. CouchDB always request document and
> his attachments with Accept: multipart/mixed header.
>
> --
> ,,,^..^,,,
>
--
-Calvin W. Metcalf
Re: CouchDB 1.5 throwing exceptions pulling attachments from
Couchbase Sync Gateway
Posted by Alexander Shorin <kx...@gmail.com>.
On Tue, Apr 22, 2014 at 9:30 PM, Calvin Metcalf
<ca...@gmail.com> wrote:
> Looks like couch
> is sending {"Accept","application/json"} in this request as well.
This request is unrelated to fetching document revisions and his
attachments, it's about fetching database information, which runs
periodically during replication. CouchDB always request document and
his attachments with Accept: multipart/mixed header.
--
,,,^..^,,,
Re: CouchDB 1.5 throwing exceptions pulling attachments from
Couchbase Sync Gateway
Posted by Calvin Metcalf <ca...@gmail.com>.
we had a similar issue with PouchDB and Couchbase Sync Gateway (
https://github.com/pouchdb/pouchdb/issues/1666) that came down to sync
gateway wanting to use multipart/related but PouchDB sending accept:
'application/json' and couchbase sending back an error. Looks like couch
is sending {"Accept","application/json"} in this request as well.
On Tue, Apr 22, 2014 at 1:23 PM, Jens Alfke <je...@couchbase.com> wrote:
> A developer is reporting that a CouchDB pull replication, with source the
> Couchbase Sync Gateway, is failing with Erlang exceptions. The cause
> appears to be a MIME multipart response from a GET of a document with
> attachments — he says “the response of SG GET
> /nam/image-97m333b8iah?revs=true&open_revs=%5B%229-9f060576f801f3c9cea5c90013c6d182%22%5D&latest=true
> properly returns a multipart document, but it seems CouchDB 1.5 is unable
> to parse the data.”
>
> I don’t think this is the known-and-recently-fixed issue with CouchDB and
> chunked encoding. Replicating attachments to CouchDB has been working for a
> while, and the developer reports "the documents only containing 2 images
> did replicate from SG to CouchDB 1.5.” It may be that the exact length or
> contents of this document are triggering some kind of edge case in CouchDB.
>
> Anyway, I can’t make head or tail of the Erlang crash logs; could someone
> skilled in the art provide some insight into what happened to CouchDB?
> Thanks!
>
> —Jens
>
> > Using CouchDB 1.3 the pull replication fails completely. Erlang throws
> some errors and completely restarts.
> >
> > Using CouchDB 1.5 the pull replication does not fail, but is never
> completed. It seems CouchDB is unable to parse the multipart response.
> > CouchDB 1.5 logs:
> >
> > [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17159.1>] Replication
> crashing because GET
> http://192.168.1.2:4985/nam/image-97m333b8iah?revs=true&open_revs=%5B%229-9f060576f801f3c9cea5c90013c6d182%22%5D&latest=truefailed
> > [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17142.1>] ** Generic server
> <0.17142.1> terminating
> > ** Last message in was {'EXIT',<0.17159.1>,kaboom}
> > ** When Server state == {state,<0.17123.1>,<0.17143.1>,20,
> > {httpdb,"http://192.168.1.2:4985/nam/",nil,
> > [{"Accept","application/json"},
> > {"User-Agent","CouchDB/1.5.0"}],
> > 30000,
> > [{socket_options,
> >
> [{keepalive,true},{nodelay,false}]}],
> > 10,250,<0.17124.1>,20},
> > {db,<0.17131.1>,<0.17132.1>,nil,
> > <<"1398170695581111">>,<0.17133.1>,
> > <0.17129.1>,<0.17135.1>,
> > {db_header,6,9,0,
> > {2230287,{3,6,2227482},1191},
> > {2228708,9,508},
> > {2232424,[],807},
> > 0,nil,nil,1000},
> > 9,
> > {btree,<0.17129.1>,
> > {2230287,{3,6,2227482},1191},
> > #Fun<couch_db_updater.10.75275056>,
> > #Fun<couch_db_updater.11.75275056>,
> > #Fun<couch_btree.5.71351651>,
> >
> #Fun<couch_db_updater.12.75275056>,snappy},
> > {btree,<0.17129.1>,
> > {2228708,9,508},
> > #Fun<couch_db_updater.13.75275056>,
> > #Fun<couch_db_updater.14.75275056>,
> > #Fun<couch_btree.5.71351651>,
> >
> #Fun<couch_db_updater.15.75275056>,snappy},
> > {btree,<0.17129.1>,
> > {2232424,[],807},
> > #Fun<couch_btree.3.71351651>,
> > #Fun<couch_btree.4.71351651>,
> >
> #Fun<couch_btree.5.71351651>,nil,snappy},
> > 9,<<"snagtracker-140417-tenpost">>,
> >
> "/usr/local/var/lib/couchdb/snagtracker-140417-tenpost.couch",
> > [],[],nil,
> > {user_ctx,null,[],undefined},
> > nil,1000,
> >
> [before_header,after_header,on_file_open],
> >
> [{user_ctx,{user_ctx,null,[],undefined}}],
> > snappy,nil,nil},
> > [<0.17159.1>],
> > nil,nil,
> > {<0.17143.1>,#Ref<0.0.2.127076>},
> > {rep_stats,1,1,0,0,0},
> > nil,<0.17144.1>,
> > {batch,[],0}}
> > ** Reason for termination ==
> > ** {process_died,<0.17159.1>,kaboom}
> >
> > [Tue, 22 Apr 2014 13:40:42 GMT] [error] [<0.17142.1>]
> {error_report,<0.17006.1>,
> > {<0.17142.1>,crash_report,
> > [[{initial_call,
> >
> {couch_replicator_worker,init,['Argument__1']}},
> > {pid,<0.17142.1>},
> > {registered_name,[]},
> > {error_info,
> > {exit,
> > {process_died,<0.17159.1>,kaboom},
> > [{gen_server,terminate,6,
> > [{file,"gen_server.erl"},{line,744}]},
> > {proc_lib,init_p_do_apply,3,
> > [{file,"proc_lib.erl"},{line,239}]}]}},
> > {ancestors,
> > [<0.17123.1>,couch_replicator_job_sup,
> > couch_primary_services,couch_server_sup,
> > <0.17007.1>]},
> > {messages,[{'EXIT',<0.17144.1>,normal}]},
> > {links,[<0.17143.1>,<0.17123.1>]},
> > {dictionary,
> > [{last_stats_report,{1398,170695,606206}}]},
> > {trap_exit,true},
> > {status,running},
> > {heap_size,1598},
> > {stack_size,27},
> > {reductions,305}],
> > [{neighbour,
> > [{pid,<0.17143.1>},
> > {registered_name,[]},
> > {initial_call,{erlang,apply,2}},
> > {current_function,{gen,do_call,4}},
> > {ancestors,[]},
> > {messages,[]},
> > {links,[<0.17142.1>]},
> > {dictionary,[]},
> > {trap_exit,false},
> > {status,waiting},
> > {heap_size,1598},
> > {stack_size,22},
> > {reductions,249}]}]]}}
>
--
-Calvin W. Metcalf