[jira] [Commented] (STORM-618) Kafka spout should provide optional way to implement at-most once semantic

["Sriharsha Chintalapani (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270321#comment-14270321 ] 

Sriharsha Chintalapani commented on STORM-618:
----------------------------------------------

[~sweetest_sj] I don't think this is valid bug. If you are not using trident i.e you are using core spout or bolt in a topology it means at-most once semantic. In your example if a user wants to avoid the replays if there is a failure in downstream bolt they can just do ack(tuple) at the top of execute method. This will ack the tuple immediately even if there is a failure after . 

> Kafka spout should provide optional way to implement at-most once semantic
> --------------------------------------------------------------------------
>
>                 Key: STORM-618
>                 URL: https://issues.apache.org/jira/browse/STORM-618
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka
>    Affects Versions: 0.9.3
>            Reporter: Adrian Seungjin Lee
>
> While it's nice for kafka spout to push failed tuple back into a sorted set and try to process it again, this way of guaranteed message processing sometimes makes situation pretty bad when a failed tuple repeatedly fails in downstream bolts since PartitionManager#fill method tries to fetch from that offset repeatedly.
> This is a corresponding code snippet.
>     private void fill() {
> ...
>         if (had_failed) {
>             offset = failed.first();
>         } else {
>             offset = _emittedToOffset;
>         }
> ...
>             msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition, offset);
> ...
> So there should be an option for a developer to decide if he wants to process failed tuple again or just skip failed tuple. One of the best thing of Storm is that spout together with trident can be implemented to guarantee at-least-once,exactly-once and at-most-once message processing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)