You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Greg Chatham (JIRA)" <ji...@apache.org> on 2019/05/16 20:50:00 UTC
[jira] [Updated] (STORM-3393) OffsetManager doesn't recover after
missing offsets
[ https://issues.apache.org/jira/browse/STORM-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Chatham updated STORM-3393:
--------------------------------
Description:
When missing offsets are encountered, but a committable offset exists after the missing offset, the condition is detected and logged but not properly processed. You will see three log messages in this case:
{code:java}
Processed non-sequential offset. The earliest uncommitted offset is no longer part of the topic. Missing offset: [{}], Processed: [{}]
...
Found committable offset: [{}] after missing offset: [{}], skipping to the committable offset
...
Topic-partition [{}] has no offsets ready to be committed{code}
However, this is not the proper handling. While a committable offset has been found, the found flag is not set to true (resulting in the 3rd log message).
The fix is to add a found=true within this logic:
In OffsetManager.java
{code:java}
if (nextEmittedOffset != null && currOffset == nextEmittedOffset) {
LOG.debug("Found committable offset: [{}] after missing offset: [{}], skipping to the committable offset",
currOffset, nextCommitOffset);
nextCommitOffset = currOffset + 1;
found = true; // ADD THIS LINE TO FIX THIS BUG
}{code}
This results in offsets not being committed, although processing will continue
was:
When missing offsets are encountered, but a committable offset exists after the missing offset, the condition is detected and logged but not properly processed. You will see three log messages in this case:
{code:java}
Processed non-sequential offset. The earliest uncommitted offset is no longer part of the topic. Missing offset: [{}], Processed: [{}]
Found committable offset: [{}] after missing offset: [{}], skipping to the committable offset
Topic-partition [{}] has no offsets ready to be committed{code}
However, this is not the proper handling. While a committable offset has been found, the found flag is not set to true (resulting in the 3rd log message).
The fix is to add a found=true within this logic:
In OffsetManager.java
{code:java}
if (nextEmittedOffset != null && currOffset == nextEmittedOffset) {
LOG.debug("Found committable offset: [{}] after missing offset: [{}], skipping to the committable offset",
currOffset, nextCommitOffset);
nextCommitOffset = currOffset + 1;
found = true; // ADD THIS LINE TO FIX THIS BUG
}{code}
This results in offsets not being committed, although processing will continue
> OffsetManager doesn't recover after missing offsets
> ---------------------------------------------------
>
> Key: STORM-3393
> URL: https://issues.apache.org/jira/browse/STORM-3393
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-client
> Affects Versions: 1.1.0, 1.1.3
> Reporter: Greg Chatham
> Priority: Major
>
> When missing offsets are encountered, but a committable offset exists after the missing offset, the condition is detected and logged but not properly processed. You will see three log messages in this case:
> {code:java}
> Processed non-sequential offset. The earliest uncommitted offset is no longer part of the topic. Missing offset: [{}], Processed: [{}]
> ...
> Found committable offset: [{}] after missing offset: [{}], skipping to the committable offset
> ...
> Topic-partition [{}] has no offsets ready to be committed{code}
> However, this is not the proper handling. While a committable offset has been found, the found flag is not set to true (resulting in the 3rd log message).
> The fix is to add a found=true within this logic:
> In OffsetManager.java
> {code:java}
> if (nextEmittedOffset != null && currOffset == nextEmittedOffset) {
> LOG.debug("Found committable offset: [{}] after missing offset: [{}], skipping to the committable offset",
> currOffset, nextCommitOffset);
> nextCommitOffset = currOffset + 1;
> found = true; // ADD THIS LINE TO FIX THIS BUG
> }{code}
> This results in offsets not being committed, although processing will continue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)