[jira] [Assigned] (STORM-2648) Kafka spout can't show acks/fails and complete latency when auto commit is enabled

["Stig Rohde Døssing (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stig Rohde Døssing reassigned STORM-2648:
-----------------------------------------

    Assignee: Stig Rohde Døssing

> Kafka spout can't show acks/fails and complete latency when auto commit is enabled
> ----------------------------------------------------------------------------------
>
>                 Key: STORM-2648
>                 URL: https://issues.apache.org/jira/browse/STORM-2648
>             Project: Apache Storm
>          Issue Type: New Feature
>          Components: storm-kafka-client
>    Affects Versions: 2.0.0, 1.1.0, 1.2.0
>            Reporter: Stig Rohde Døssing
>            Assignee: Stig Rohde Døssing
>            Priority: Minor
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> The storm-kafka-client spout currently emits tuples with no message ids if auto commit is enabled. This causes the ack/fail/complete latency counters in Storm UI to be 0. In some cases this is desirable because the user may not care, and doesn't want the overhead of Storm tracking tuples. [~avermeerbergen] expressed a desire to be able to use auto commit without these counters being disabled, presumably to monitor topology performance.
> In order to support having acks/fails and complete latency when using auto commit, we need to make the spout always emit tuples with anchors. Since the spout doesn't care about acks or fails when auto commit is enabled, we should make the spout ack/fail methods return immediately if auto commit is enabled.
> I think we don't lose much by switching to this approach. Users that want the statistics in Storm UI can enable auto commit and leave topology ackers at some non-zero number. Users that don't care and don't want the overhead of having Storm track the tuples can set topology ackers to 0, which should make the spout behave a lot like it does now. 
> The only use case I can think of where this won't work is if someone with multiple spouts in a topology needs one to be auto commit and one to be at-least-once, and they can't live with the overhead of tuple tracking for the auto commit spout. If this is a real use case, it can be worked around with an extra configuration parameter to switch whether tuples are emitted unanchored, but I'd rather keep it simple for now.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)