[jira] [Commented] (STORM-495) Add delayed retries to KafkaSpout

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224126#comment-14224126 ] 

ASF GitHub Bot commented on STORM-495:
--------------------------------------

Github user rick-kilgore commented on the pull request:

    https://github.com/apache/storm/pull/254#issuecomment-64317893
  
    This PR is now update to incorporate most of the changes suggested by @wurstmeister.  I also discovered that the conflict I was worried about with TOPOLOGY_MESSAGE_TIMEOUT_SECS does not exist (this timeout affects life of a tuple between retries, not across them), so I removed the special handling and warning message logic.
    
    I also up-merged and created tests for the new ExponentialBackoffMsgRetryManager class.


> Add delayed retries to KafkaSpout
> ---------------------------------
>
>                 Key: STORM-495
>                 URL: https://issues.apache.org/jira/browse/STORM-495
>             Project: Apache Storm
>          Issue Type: Improvement
>    Affects Versions: 0.9.3
>         Environment: all environments
>            Reporter: Rick Kilgore
>            Priority: Minor
>              Labels: kafka, retry
>
> If a tuple in the topology originates from the KafkaSpout from the external/storm-kafka sources, and if a bolt in the topology indicates a failure by calling fail() on its OutputCollector, the KafkaSpout will immediately retry the message.
> We wish to use this failure and retry behavior in our ingestion system whenever we experience a recoverable error from a downstream system, such as a 500 or 503 error from a service we depend on.  But with the current KafkaSpout behavior, doing so results in a tight loop where we retry several times over a few seconds and then give up.  I want to be able to delay retry to give the downstream service some time to recover.  Ideally, I would like to have configurable, exponential backoff retry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)