[jira] [Commented] (STORM-495) Add delayed retries to KafkaSpout

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14129165#comment-14129165 ] 

ASF GitHub Bot commented on STORM-495:
--------------------------------------

GitHub user rick-kilgore opened a pull request:

    https://github.com/apache/incubator-storm/pull/254

    [STORM-495] KafkaSpout retries with exponential backoff

    This is an implementation of the improvement request STORM-495.
    
    The KafkaSpout keeps track of failed messages through multiple retries, and delays their resubmission to the topology with an exponentially increasing delay time.  The spout does not terminate the retry cycle (it is my conviction that it should not do so, because it cannot report context about the failure that happened to abort the reqeust), it only handles delaying the retries.  A bolt in the topology is still expected to eventually call ack() instead of fail() to stop the cycle.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HBOCodeLabs/incubator-storm retries

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-storm/pull/254.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #254
    
----
commit cac70e77d3417e13082495aaac126f25a6d07eff
Author: Rick Kilgore <ri...@hbo.com>
Date:   2014-08-23T04:49:08Z

    exponential backoff for failed messages

commit f9d2948a9e2dadbbb99128fd6570038e2096eabd
Author: Rick Kilgore <ri...@hbo.com>
Date:   2014-09-03T22:04:32Z

    Merge branch 'master' into retryDelay

commit 6138e945894c2e0b95666909af51fd336862ce6d
Author: Rick Kilgore <ri...@hbo.com>
Date:   2014-09-06T06:57:17Z

    Merge branch 'master' into retryDelay

commit 1c748a7fde11ebc7d209c16ad2b30bf19a316395
Author: Rick Kilgore <ri...@hbo.com>
Date:   2014-09-06T07:08:12Z

    Merge pull request #1 from rick-kilgore/retryDelay
    
    exponential backoff for failed messages
    moving repo to HBOCodeLabs

commit 04b64bc66d5c29ba77c455f0125200ed8335376f
Author: rick-kilgore <ri...@gmail.com>
Date:   2014-09-08T20:49:18Z

    add comment and different default for retry

commit 05336764408921f3ebcc4ac5b3e82da10e713bea
Author: rick-kilgore <ri...@gmail.com>
Date:   2014-09-08T20:58:01Z

    default settings = no retry delay
    
    default settings = no retry delay - i.e., preserve legacy behavior

commit 79e3efe2558d252f2122facdb0ce5e69c9c62104
Author: Rick Kilgore <ri...@hbo.com>
Date:   2014-09-09T21:52:37Z

    make config names more consistent with each other

commit 2aa03468db3971a11d07048628908074f2a0d3b3
Author: Rick Kilgore <ri...@hbo.com>
Date:   2014-09-09T23:03:34Z

    do the time calc only once
    
    do the due time calculation for a MessageRetryRecord when one is
    created, not every time isReadyForRetry() is calculated

commit f4fa76c63efefa987ba124dd7dc8f895f9fe6dcc
Author: Rick Kilgore <ri...@hbo.com>
Date:   2014-09-09T23:06:56Z

    silly fix
    
    use member var, don't pass it

----


> Add delayed retries to KafkaSpout
> ---------------------------------
>
>                 Key: STORM-495
>                 URL: https://issues.apache.org/jira/browse/STORM-495
>             Project: Apache Storm (Incubating)
>          Issue Type: Improvement
>    Affects Versions: 0.9.3-incubating
>         Environment: all environments
>            Reporter: Rick Kilgore
>            Priority: Minor
>              Labels: kafka, retry
>
> If a tuple in the topology originates from the KafkaSpout from the external/storm-kafka sources, and if a bolt in the topology indicates a failure by calling fail() on its OutputCollector, the KafkaSpout will immediately retry the message.
> We wish to use this failure and retry behavior in our ingestion system whenever we experience a recoverable error from a downstream system, such as a 500 or 503 error from a service we depend on.  But with the current KafkaSpout behavior, doing so results in a tight loop where we retry several times over a few seconds and then give up.  I want to be able to delay retry to give the downstream service some time to recover.  Ideally, I would like to have configurable, exponential backoff retry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)