[jira] [Commented] (COUCHDB-1583) improve the flow control in ibrowse streams

["Adam Kocoloski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/COUCHDB-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488134#comment-13488134 ] 

Adam Kocoloski commented on COUCHDB-1583:
-----------------------------------------

Thanks for opening this ticket Bob.  Sorry for not firing off a topic branch sooner.  When we started using the new replicator internally at Cloudant we noticed that long-running replications would ultimately consume a ton of memory on the node mediating the replication and crash.  I tracked it down to the mailbox for the process consuming the _changes feed over HTTP, and from there to a misuse of ibrowse's stream_next API.  Admittedly the API is super-confusing; it should have a big read warning box telling you that you may receive multiple messages for each invocation of the function.

The patch is working well for us, and given that the commit landed directly on master I'm going to go ahead and resolve this one.
                
> improve the flow control in ibrowse streams
> -------------------------------------------
>
>                 Key: COUCHDB-1583
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1583
>             Project: CouchDB
>          Issue Type: Improvement
>          Components: Replication
>            Reporter: Bob Dionne
>            Assignee: Adam Kocoloski
>             Fix For: 1.3
>
>
> The flow control in ibrowse's async response streams is tricky.  We call
> stream_next to pull more data off the socket, but it seems that ibrowse
> will sometimes split that data into multiple messages.  If we call
> stream_next for each message we process we end up with an overflowing
> mailbox.
> This commit[1] changes the consumer so that it clears out the mailbox before
> calling stream_next.
> [1] 8ccf696f833a0f0a453d

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira