You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Stig Rohde Døssing (JIRA)" <ji...@apache.org> on 2017/10/12 19:51:00 UTC
[jira] [Commented] (STORM-2430) Potential Race condition in Kafka
Spout
[ https://issues.apache.org/jira/browse/STORM-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202521#comment-16202521 ]
Stig Rohde Døssing commented on STORM-2430:
-------------------------------------------
Sorry [~hmclouro], I missed this issue somehow. I think this will be completely fixed with https://issues.apache.org/jira/browse/STORM-2549 and https://issues.apache.org/jira/browse/STORM-2546.
> Potential Race condition in Kafka Spout
> ---------------------------------------
>
> Key: STORM-2430
> URL: https://issues.apache.org/jira/browse/STORM-2430
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
> Affects Versions: 1.0.2, 1.x
> Reporter: Kushal Bhatt
>
> Kafka spout hangs when the number of uncommitted messages exceeds the max allowed uncommitted messages and some intermediate tuples have failed in down stream bolt.
> Steps of reproduction.
> Create a simple topology with one kafka spout and a slow bolt.
> In kafka spout set the maximum uncommitted messages to a small number like 100.
> Bolt should process 10 tuples in second. And program it to fail on some random tuples. For eg: say tuple number 10 fails. Also assume there is only 1 Kafka partition the spout reads from.
> Spout on first execution of nextTuple() gets 110 records and emits them. At this point number of uncommitted message would be 110.
> First 9 tuples are acked by the bolt. 10th tuple is failed by the bolt. KafkaSpout puts it on retry queue.
> Tuple number 11 to 110 are acked by bolt . But spout only commits till offset 9.[link | https://github.com/apache/storm/blob/1.0.x-branch/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L510]
> Now, the number of uncommitted messages = 110 - 9 = 101 > 100 (max allowed uncommitted messages)
> No new records are polled from kafka.[link | https://github.com/apache/storm/blob/1.0.x-branch/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L239]. The spout is stuck as the nothing is polled.
> Solution is to explicitly go through retry queue explicitly and emit tuples that are ready on every nextTuple().
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)