[jira] [Updated] (STORM-2639) Kafka Spout incorrectly computes numCommittedOffsets due to voids in the topic (topic compaction)

["Prasanna Ranganathan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prasanna Ranganathan updated STORM-2639:
----------------------------------------
    Description: 
This is a followup to STORM-2505 to fix a minor issue with the computation of numUncommittedOffsets in OffsetManager. 


  was:
Kafka maintains the spout progress (offsets for partitions) which can hold a value which no longer exists (or offset+1 doesn't exist) in the topic due to following reasons
* Topology stopped processing (or died) & topic got compacted (cleanup.policy=compact) leaving offset voids in the topic.
* Topology stopped processing (or died) & Topic got cleaned up (cleanup.policy=delete) and the offset.

When the topology starts processing again (or restarted), the spout logic suggests that the next offset has to be (committedOffset+1) for the spout to make progress, which will never be the case as (committedOffset+1) has been removed from the topic and will never be acked.

{code:title=OffsetManager.java|borderStyle=solid}
 if (currOffset == nextCommitOffset + 1) {            // found the next offset to commit
      found = true;
      nextCommitMsg = currAckedMsg;
      nextCommitOffset = currOffset;
} else if (currOffset > nextCommitOffset + 1) {
      LOG.debug("topic-partition [{}] has non-continuous offset [{}]. It will be processed in a subsequent batch.", tp, currOffset);
}
{code}

A smart forwarding mechanism has to be built so as to forward the spout pivot to the next logical location, instead of a hardcoded single forward operation.



> Kafka Spout incorrectly computes numCommittedOffsets due to voids in the topic (topic compaction)
> -------------------------------------------------------------------------------------------------
>
>                 Key: STORM-2639
>                 URL: https://issues.apache.org/jira/browse/STORM-2639
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka-client
>    Affects Versions: 1.x
>            Reporter: Prasanna Ranganathan
>            Assignee: Vivek Mittal
>             Fix For: 2.0.0, 1.1.1, 1.2.0
>
>
> This is a followup to STORM-2505 to fix a minor issue with the computation of numUncommittedOffsets in OffsetManager. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)