You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@storm.apache.org by "Stig Rohde Døssing (JIRA)" <ji...@apache.org> on 2018/12/08 21:09:00 UTC

[jira] [Updated] (STORM-3301) The KafkaSpout can in some cases still replay tuples that were already committed

     [ https://issues.apache.org/jira/browse/STORM-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stig Rohde Døssing updated STORM-3301:
--------------------------------------
    Description: 
In the fix for STORM-2666 and followups, we added logic to handle cases where the spout received the ack for an offset after the following offsets were already acked. The issue was that the spout might commit all the acked offsets, but not adjust the consumer position forward, or clear out waitingToEmit properly. If the acked offset was sufficiently far behind the log end offset, the spout might end up polling for offsets it had already committed.

The fix is slightly wrong. When the consumer position drops behind the committed offset

> The KafkaSpout can in some cases still replay tuples that were already committed
> --------------------------------------------------------------------------------
>
>                 Key: STORM-3301
>                 URL: https://issues.apache.org/jira/browse/STORM-3301
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka-client
>    Affects Versions: 2.0.0, 1.2.2
>            Reporter: Stig Rohde Døssing
>            Assignee: Stig Rohde Døssing
>            Priority: Major
>
> In the fix for STORM-2666 and followups, we added logic to handle cases where the spout received the ack for an offset after the following offsets were already acked. The issue was that the spout might commit all the acked offsets, but not adjust the consumer position forward, or clear out waitingToEmit properly. If the acked offset was sufficiently far behind the log end offset, the spout might end up polling for offsets it had already committed.
> The fix is slightly wrong. When the consumer position drops behind the committed offset



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)