[jira] [Created] (STORM-2107) Storm-kafka-client: Slow topology start breaks the Kafka Client with session timeouts

["Jeff Fenchel (JIRA)" <ji...@apache.org>]

Jeff Fenchel created STORM-2107:
-----------------------------------

             Summary: Storm-kafka-client: Slow topology start breaks the Kafka Client with session timeouts 
                 Key: STORM-2107
                 URL: https://issues.apache.org/jira/browse/STORM-2107
             Project: Apache Storm
          Issue Type: Bug
            Reporter: Jeff Fenchel


Specifically CommitFailedExceptions are being thrown when an attempt to commit the offset is made. The problem is detailed here: 

https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records

On topology start, the spout will start emitting tuples before all bolts are prepared. This means that tuples will hang around in the topology until everything is ready to start processing. This caused the max spout pending limit to be hit, and the spout to stop polling kafka for the duration of the timeout. It seems like there could be other use cases that cause this too. 

I propose we start polling on a timer kafka to avoid this.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)