You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "FrankChen021 (via GitHub)" <gi...@apache.org> on 2026/02/13 07:25:40 UTC

[PR] Improve offset auto reset (druid)

FrankChen021 opened a new pull request, #18301:
URL: https://github.com/apache/druid/pull/18301

   Fixes #18282
   
   
   ### Description
   
   Improve the auto reset logic as proposed in the above issue:
   
   1. only supervisor is responsible for auto reset. task never triggers the auto reset flow
   2. task always throws `OffsetOutOfRangeException` if this exception is raised, but will publish ingested message under such case
   3. supervisor resets offset to earliest position if it detects offset of a particular partition is unavailable
   4. supervisor starts task more quickly if it needs to reset offsets
   5. update the obscure error message like `DataSourceMetadata is not found while reset` in task report to make it intuitive when supervisor kills a task because of offset reset
   
   ### NOTE
   this relies on #18226, which publishes segments when OffsetOutOfRangeException is raised.
   
   
   
   This PR has:
   
   - [X] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [X] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18301:
URL: https://github.com/apache/druid/pull/18301#issuecomment-3417559630

   This pull request/issue has been closed due to lack of activity. If you think that
   is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18301: Improve offset auto reset
URL: https://github.com/apache/druid/pull/18301


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #18301: Improve offset auto reset
URL: https://github.com/apache/druid/pull/18301


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18301:
URL: https://github.com/apache/druid/pull/18301#issuecomment-3314279748

   This pull request has been marked as stale due to 60 days of inactivity.
   It will be closed in 4 weeks if no further activity occurs. If you think
   that's incorrect or this pull request should instead be reviewed, please simply
   write any comment. Even if closed, you can still revive the PR at any time or
   discuss it on the dev@druid.apache.org list.
   Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18301:
URL: https://github.com/apache/druid/pull/18301#issuecomment-4248320652

   This pull request has been marked as stale due to 60 days of inactivity.
   It will be closed in 4 weeks if no further activity occurs. If you think
   that's incorrect or this pull request should instead be reviewed, please simply
   write any comment. Even if closed, you can still revive the PR at any time or
   discuss it on the dev@druid.apache.org list.
   Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18301:
URL: https://github.com/apache/druid/pull/18301#discussion_r2218552394


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -3992,31 +3999,46 @@ private OrderedSequenceNumber<SequenceOffsetType> getOffsetFromStorageForPartiti
     if (sequence != null) {
       log.debug("Getting sequence [%s] from metadata storage for partition [%s]", sequence, partition);
       if (!taskTuningConfig.isSkipSequenceNumberAvailabilityCheck()) {
-        if (!checkOffsetAvailability(partition, sequence)) {
-          if (taskTuningConfig.isResetOffsetAutomatically()) {
+        // Check if current sequence is less than or equal to earliestSequenceNumber in the stream
+        SequenceOffsetType earliestSequenceNumber = getOffsetFromStreamForPartition(partition, true);
+        boolean isOffsetAvailable = earliestSequenceNumber != null
+                                    && makeSequenceNumber(sequence).isAvailableWithEarliest(makeSequenceNumber(earliestSequenceNumber));
+
+        if (!isOffsetAvailable) {
+          if (taskTuningConfig.isResetOffsetAutomatically() && earliestSequenceNumber != null) {
             resetInternal(
-                createDataSourceMetaDataForReset(ioConfig.getStream(), ImmutableMap.of(partition, sequence))
+                createDataSourceMetaDataForReset(ioConfig.getStream(),
+                                                 // Reset to the earliest offset for the partition
+                                                 ImmutableMap.of(partition, earliestSequenceNumber)),
+                // auto reset flag
+                true
             );
-            throw new StreamException(
-                new ISE(
-                    "Previous sequenceNumber [%s] is no longer available for partition [%s] - automatically resetting"
-                    + " sequence",
-                    sequence,
-                    partition
-                )
+            log.makeAlert("Offsets were reset automatically, potential data duplication or loss")
+               .addData("dataSource", this.dataSource)
+               .addData("partition", partition)
+               .addData("offset", sequence.toString())
+               .addData("newOffset", earliestSequenceNumber.toString())
+               .emit();
+
+            // Return the earliest offset as the new sequence number
+            return makeSequenceNumber(
+                earliestSequenceNumber,
+                useExclusiveStartSequenceNumberForNonFirstSequence()
             );
           } else {
             throw new StreamException(
                 new ISE(
-                    "Previous sequenceNumber [%s] is no longer available for partition [%s]. You can clear the previous"
+                    "Previous sequenceNumber [%s] is no longer available for partition [%s] which now has the least sequence number [%s]. You can clear the previous"
                     + " sequenceNumber and start reading from a valid message by using the supervisor's reset API.",
-                    sequence,
+                    sequence.toString(),
+                    earliestSequenceNumber == null ? "null" : earliestSequenceNumber.toString(),
                     partition

Review Comment:
   The error message construction places the `earliestSequenceNumber` parameter in the wrong position. Based on the message format, it should be: `sequence.toString(), partition, earliestSequenceNumber == null ? "null" : earliestSequenceNumber.toString()`
   ```suggestion
                       partition,
                       earliestSequenceNumber == null ? "null" : earliestSequenceNumber.toString()
   ```



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -1900,7 +1903,8 @@ public void resetInternal(DataSourceMetadata dataSourceMetadata)
         if (currentMetadata == null) {
           metadataUpdateSuccess = true;
         } else {
-          final DataSourceMetadata newMetadata = currentMetadata.minus(resetMetadata);
+          final DataSourceMetadata newMetadata = currentMetadata.plus(resetMetadata);

Review Comment:
   This line changes the logic from `minus(resetMetadata)` to `plus(resetMetadata)`, which fundamentally alters how metadata is combined during reset operations. This appears to be a breaking change that could cause incorrect offset handling. Please verify this change is intentional and correct.
   ```suggestion
             final DataSourceMetadata newMetadata = currentMetadata.minus(resetMetadata);
   ```



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:
##########
@@ -3992,31 +3999,46 @@ private OrderedSequenceNumber<SequenceOffsetType> getOffsetFromStorageForPartiti
     if (sequence != null) {
       log.debug("Getting sequence [%s] from metadata storage for partition [%s]", sequence, partition);
       if (!taskTuningConfig.isSkipSequenceNumberAvailabilityCheck()) {
-        if (!checkOffsetAvailability(partition, sequence)) {
-          if (taskTuningConfig.isResetOffsetAutomatically()) {
+        // Check if current sequence is less than or equal to earliestSequenceNumber in the stream
+        SequenceOffsetType earliestSequenceNumber = getOffsetFromStreamForPartition(partition, true);
+        boolean isOffsetAvailable = earliestSequenceNumber != null
+                                    && makeSequenceNumber(sequence).isAvailableWithEarliest(makeSequenceNumber(earliestSequenceNumber));
+
+        if (!isOffsetAvailable) {
+          if (taskTuningConfig.isResetOffsetAutomatically() && earliestSequenceNumber != null) {
             resetInternal(
-                createDataSourceMetaDataForReset(ioConfig.getStream(), ImmutableMap.of(partition, sequence))
+                createDataSourceMetaDataForReset(ioConfig.getStream(),
+                                                 // Reset to the earliest offset for the partition
+                                                 ImmutableMap.of(partition, earliestSequenceNumber)),
+                // auto reset flag
+                true
             );
-            throw new StreamException(
-                new ISE(
-                    "Previous sequenceNumber [%s] is no longer available for partition [%s] - automatically resetting"
-                    + " sequence",
-                    sequence,
-                    partition
-                )
+            log.makeAlert("Offsets were reset automatically, potential data duplication or loss")
+               .addData("dataSource", this.dataSource)
+               .addData("partition", partition)
+               .addData("offset", sequence.toString())
+               .addData("newOffset", earliestSequenceNumber.toString())
+               .emit();
+
+            // Return the earliest offset as the new sequence number
+            return makeSequenceNumber(
+                earliestSequenceNumber,
+                useExclusiveStartSequenceNumberForNonFirstSequence()
             );
           } else {
             throw new StreamException(
                 new ISE(
-                    "Previous sequenceNumber [%s] is no longer available for partition [%s]. You can clear the previous"
+                    "Previous sequenceNumber [%s] is no longer available for partition [%s] which now has the least sequence number [%s]. You can clear the previous"

Review Comment:
   The error message refers to 'least sequence number' but should use 'earliest sequence number' to be consistent with the variable name `earliestSequenceNumber` used in the code.
   ```suggestion
                       "Previous sequenceNumber [%s] is no longer available for partition [%s] which now has the earliest sequence number [%s]. You can clear the previous"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "jtuglu1 (via GitHub)" <gi...@apache.org>.
jtuglu1 commented on PR #18301:
URL: https://github.com/apache/druid/pull/18301#issuecomment-4401851680

   @FrankChen021 any thoughts here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18301:
URL: https://github.com/apache/druid/pull/18301#issuecomment-4910323243

   This pull request has been marked as stale due to 60 days of inactivity.
   It will be closed in 4 weeks if no further activity occurs. If you think
   that's incorrect or this pull request should instead be reviewed, please simply
   write any comment. Even if closed, you can still revive the PR at any time or
   discuss it on the dev@druid.apache.org list.
   Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Improve offset auto reset (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18301:
URL: https://github.com/apache/druid/pull/18301#issuecomment-5199026103

   This pull request/issue has been closed due to lack of activity. If you think that
   is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org