You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@storm.apache.org by "Xu Mingmin (JIRA)" <ji...@apache.org> on 2017/02/03 18:42:51 UTC

[jira] [Created] (STORM-2340) KafkaSpout is blocked in AutoCommitMode

Xu Mingmin created STORM-2340:
---------------------------------

             Summary: KafkaSpout is blocked in AutoCommitMode
                 Key: STORM-2340
                 URL: https://issues.apache.org/jira/browse/STORM-2340
             Project: Apache Storm
          Issue Type: Improvement
          Components: storm-kafka-client
            Reporter: Xu Mingmin


What's the issue?
When Storm topology is run on 'At-Most-Once' mode, with ack executor=0, KafkaSpout cannot emit records after some time.

What's the cause?
Function poll() controls whether it need to pull more data from Kafka cluster. With the condition !waitingToEmit() && numUncommittedOffsets < maxUncommittedOffsets, it's always FALSE after reaching threshold, as numUncommittedOffsets is creasing and never reset on 'At-Most-Once' mode.

What's the solution?
I change the condition to !waitingToEmit() && (numUncommittedOffsets < maxUncommittedOffsets || consumerAutoCommitMode), that return TRUE with AutoCommitMode regarding of UnCommittedOffset, see the comments inline.
Also, it's not required to track emitted(msgId) and numUncommittedOffsets with AutoCommitMode .



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)