[GitHub] calonso commented on issue #745: Replication with attachments never completes, {mp_parser_died,noproc} error

[gi...@git.apache.org]

calonso commented on issue #745: Replication with attachments never completes, {mp_parser_died,noproc} error
URL: https://github.com/apache/couchdb/issues/745#issuecomment-336514161
 
 
   Hi everyone!!
   
   I think I have some more information on this issue in the form of a side effect. My setup is a small cluster, with just 3 nodes continuously replicating a few databases from another, bigger one. Only 3 databases out of all the ones being replicated hold attachments and, by chance, the same node is responsible for replicating the 3 of them. That node throws the described error quite often (a few thousand times per hour), depending on the speed at which documents are received.
   
   That particular node shows a continuous increment on the `process_count` metric read from the `_system` endpoint. Growing at a similar rate of this errors' rate. That metric grows from about 1.2k processes that the nodes start with up to a bit above 5k when it gets *frozen*. It stops responding on the clustered (5984) endpoint and doesn't replicate any more data. But annoyingly it is not considered as down in the cluster, so the other nodes are not taking his responsibilities over.
   
   After connecting the Observer to that node, to see which processes are there I could see a lot of `erlang:apply/2` in function `couch_httpd_multipart:maybe_send_data/1` with 0 reductions and 0 messages in the queue and also a lot of `mochiweb_acceptor:init/4` in function `couch_doc:-doc_from_multi_part_stream/3-fun-1-/1` Some of them with 1 message on the queue, some of them with 0 and 0 reductions as well...
   
   Also this node has quite many 'erlang:apply/2' processes in function 'couch_http_multipart:mp_parse_attrs/2'.
   
   I think there may be something preventing the processes from exiting and that's why they pile up until it freezes.
   
   Hope this helps.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services