[jira] [Commented] (STORM-886) Automatic Back Pressure

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14939711#comment-14939711 ] 

ASF GitHub Bot commented on STORM-886:
--------------------------------------

Github user rsltrifork commented on the pull request:

    https://github.com/apache/storm/pull/700#issuecomment-144703338
  
    Does this solve the problem of tuple timeout, when a bolt is completely stalled waiting for an external component to get back up?
    I believe waiting for too long in a bolt triggers the tuple timeout, which causes the Spout to reemit, which is usually not what we want, when a bolt is waiting in a controlled manner before it can resume computation. 
    
    Ideally, the tuple timeout should be used as a last resort to detect that internal storm components don't respond. And back pressure should ensure that the Spout doesn't reemit to temporarily busy/blocked bolts - regardless of timeout.


> Automatic Back Pressure
> -----------------------
>
>                 Key: STORM-886
>                 URL: https://issues.apache.org/jira/browse/STORM-886
>             Project: Apache Storm
>          Issue Type: Improvement
>            Reporter: Robert Joseph Evans
>            Assignee: Zhuo Liu
>             Fix For: 0.11.0
>
>         Attachments: aSimpleExampleOfBackpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG since different components may have unmatched tuple processing speed. Currently, the tuples may get dropped if the downstream components can not process as quickly, thereby causing a waste of network bandwidth and processing capability. In addition, it is difficult to tune the max.spout.pending parameter for best backpressure performance. Therefore, an automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that  does not rely on acking or max spout pending.  Instead spouts throttle not only when max.spout.pending is hit, but also if any bolt has gone over a high water mark in their input queue, and has not yet gone below a low water mark again.  There is a lot of room for potential improvement here around control theory and having spouts only respond to downstream bolts backing up, but a simple bang-bang controller like this is a great start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)