[jira] [Updated] (STORM-39) Expose number of attempts for a batch within Trident API

["Rick Kellogg (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-39?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rick Kellogg updated STORM-39:
------------------------------
    Component/s: storm-core

> Expose number of attempts for a batch within Trident API
> --------------------------------------------------------
>
>                 Key: STORM-39
>                 URL: https://issues.apache.org/jira/browse/STORM-39
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-core
>            Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/352
> This would let someone implement policies like "retry a batch up to 4 times, then skip it".
> -----------------------------------------------------------------------------------------------------
> nathanmarz: 
> I think the best strategy is to just expose this in the coordinator. isReady can be changed to take in a TransactionAttempt and return an enum:
> enum BatchAction {
>    PROCESS,
>    WAIT,
>    SKIP
> }
> Perhaps isReady can be renamed to "getBatchAction".
> -----------------------------------------------------------------------------------------------------
> nathanmarz: 
> Actually the semantics aren't that clear. What does skip mean for an opaque spout, for example? In an opaque spout, not every batch is the same.
> Additionally, multiple spouts can be participating in the same batch. You'd want SKIP to cause only that one particular spout to skip the batch, while the other one emits.
> -----------------------------------------------------------------------------------------------------
> mrclip:
> It might make sense pull the failed records into a separate stream -- one that could be routed into a separate kafka topic for instance, or counted and then dropped. If nothing is listening on that stream, the records are effectively skipped. The spout just emits transparently, and the skipped records can still be transactionally handled on the Lethe stream.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)