[jira] [Created] (STORM-2957) Offsets are not committed when no ackers set (ackers=0) and KafkaSpout stops polling messages once it reaches the max uncommitted offsets

["Ishwara Varnasi (JIRA)" <ji...@apache.org>]

Ishwara Varnasi created STORM-2957:
--------------------------------------

             Summary: Offsets are not committed when no ackers set (ackers=0) and KafkaSpout stops polling messages once it reaches the max uncommitted offsets 
                 Key: STORM-2957
                 URL: https://issues.apache.org/jira/browse/STORM-2957
             Project: Apache Storm
          Issue Type: Bug
          Components: storm-kafka-client
    Affects Versions: 1.0.5
            Reporter: Ishwara Varnasi


A KafkaSpout stops polling messages from topic when topology has no ackers set ("topology.acker.executors"= 0). This happens after KafkaSpout polls (and emits) messages equal to the max uncommitted offsets (KafkaspoutConfig.maxUncommittedOffsets).

This happens because the KafkaSpout.ack() is called even before the emit method adds the message to "emitted" collection (Set). In such cases, the message is not added "acked" collection (Map), and hence never committed. This seem to be bug introduced in a version later than storm-kafka-client 1.0.1 and current code (1.0.5) has a check in ack() that checks if the acked message is available in "emitted" collection and if not, ignores it (never adds it to "acked" collection)

Steps to reproduce issue:
1. Need a topology with KafkaSpout from storm-kafka-client
2. Set number of ackers ("topology.acker.executors") to o (org.apache.storm.Config.setNumAckers(0)
3. While creating KafkaSpout instance, set SpoutConfig.setMaxUncommittedOffsets to small number (50 or 100)
4. Start topology, see to it that KafkaSpout polls more messages from Kafka that setMaxUncommittedOffsets
5. KafkaSpout stops polling messages from topic



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)