You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Pankaj260100 (via GitHub)" <gi...@apache.org> on 2023/10/12 10:40:29 UTC

[PR] Adding retries to update the metadata store instead of failure (druid)

Pankaj260100 opened a new pull request, #15141:
URL: https://github.com/apache/druid/pull/15141

   <!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
   
   <!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
   
   Fixes #15054.
   
   ### Description
   
   -  Currently, If 2 tasks are consuming from the same partitions, try to publish the segment and update the metadata, the second task can fail because the end offset stored in the metadata store doesn't match with the start offset of the second task. We can fix this by retrying instead of failing.
   - AFAIK apart from the above issue, the metadata mismatch can happen in 2 scenarios:
      
       1. when we update the input topic name for the data source
       2. when we run 2 replicas of ingestion tasks(1 replica will publish and 1 will fail as the first replica has already updated the metadata).
       
   - Implemented the comparable function to compare the last committed end offset and new Sequence start offset. And retry if the new sequence start offset is ahead of the stored end offset.
   - Added a new test case to test this kind of scenario. Where two task tries to publish for the same partition.
   - Updated the existing test case where it tries to publish {1 -> 2} and {null} stored in the metadata store. Since Now it will retry for this as well.  
   
   This PR has:
   
   - [x] been self-reviewed.
   - [x] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1444463955


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -61,4 +62,11 @@ SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
   SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> minus(
       SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other
   );
+
+  /**
+   * Compare this and the other sequence offsets using comparator.
+   * Returns 1, if this sequence is ahead of the other.

Review Comment:
   I have reverted this: the test case where a new kafka partition gets added and wants to publish for the first time will fail as it's not in the old committed offset.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1445713069


##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/BaseAppenderatorDriver.java:
##########
@@ -652,19 +656,16 @@ ListenableFuture<SegmentsAndCommitMetadata> publishInBackground(
                   segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
                 }
               } else {
+                log.errorSegments(ourSegments, "Failed to publish segments");
+                if (publishResult.getErrorMsg() != null && publishResult.getErrorMsg().contains(("Failed to update the metadata Store. The new start metadata is ahead of last commited end state."))) {
+                  throw new ISE(publishResult.getErrorMsg());
+                }
                 // Our segments aren't active. Publish failed for some reason. Clean them up and then throw an error.
                 segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);

Review Comment:
   For this, I will need to add the RetryUtils.retry() in the try/catch block, right? or is there a another way of doing this 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1375643913


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -61,4 +62,11 @@ SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
   SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> minus(
       SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other
   );
+
+  /**
+   * Compare this and the other sequence offsets using comparator.
+   * Returns 1, if this sequence is ahead of the other.

Review Comment:
   Could you please elaborate what a sequence being ahead of the other means?
   
   I think that it means that the partition-wise sequence number of the first is greater than or equal to the other's with a strict greater than condition for at least one of the partitions. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "xvrl (via GitHub)" <gi...@apache.org>.
xvrl commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1357443126


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamDataSourceMetadata.java:
##########
@@ -59,6 +59,18 @@ public boolean matches(DataSourceMetadata other)
     return plus(other).equals(other.plus(this));
   }
 
+  @Override
+  public int compareTo(DataSourceMetadata other)
+  {
+    if (!getClass().equals(other.getClass())) {
+      return -1;
+    }

Review Comment:
   this seems inconsistent with other implementations where we throw an exception when we encounter a different class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1357983261


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   Yes. Kinesis sequence offsets are treated as opaque string. 



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   This could be done. Two methods should be added to DataSourceMetadata
   ```
   isComparable (default return false)
   compare() (default return 0)
   ```
   
   These two methods will be implemented only by the KafkaDataSourceMetadata. In that class, you can parse the offset to long. 
   
   In the IndexerSQLMetadataStorageCoordinator, you would first check isComparable before calling compare method. 
   
   How does this sound? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "xvrl (via GitHub)" <gi...@apache.org>.
xvrl commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1364408234


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   how would we handle this potential failure scenario in Kinesis then? It seems that the issue is not specific to Kafka and could happen there as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1815854719

   @abhishekagarwal87, I have 30 ingestion tasks(set 1) running in the Druid cluster. At the time of task rollover, the existing 30 tasks(set 1) start publishing and the new 30(set 2) start ingesting. I submitted the supervisor config and the new 30(set 2) also start publishing. Now, there are a couple of tasks in set 2 which face this issue where the start metadata doesn't match with Metadatastored in DB(because all tasks from set 1 haven't finished updating the metadata store). Now these tasks start retrying but they were not retrying in parallel, I looked into the logs and the retry happened in sequence. Like task A from set 2 completes all 10 retries and then task B from set 2 completes all 10 retries and then other tasks which are facing this issue. During this, the overlord is not doing anything apart from retrying and after all tasks from set 2 are finished retrying the tasks from set 1 updates the metadata store. But all this should happen in parallel, right? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 merged PR #15141:
URL: https://github.com/apache/druid/pull/15141


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1375646836


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +149,37 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other, Comparator<SequenceOffsetType> comparator)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);

Review Comment:
   Does this need to change slightly according to https://github.com/apache/druid/pull/15141/files#r1375643913?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1382817405


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -61,4 +62,11 @@ SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
   SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> minus(
       SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other
   );
+
+  /**
+   * Compare this and the other sequence offsets using comparator.
+   * Returns 1, if this sequence is ahead of the other.

Review Comment:
   @AmatyaAvadhanula, We will only retry when the first is greater than the other's; in case both are equal, this will not fail to publish, So no point in retry, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1445865193


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,38 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other, Comparator<SequenceOffsetType> comparator)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      boolean res = false;
+      for (Map.Entry<PartitionIdType, SequenceOffsetType> entry : partitionSequenceNumberMap.entrySet()) {
+        PartitionIdType partitionId = entry.getKey();
+        SequenceOffsetType sequenceOffset = entry.getValue();
+        if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && comparator.compare(sequenceOffset, otherStart.partitionSequenceNumberMap.get(partitionId)) > 0) {

Review Comment:
   Would startMetadataMatchesExisting be true if the entire metadata is null or if any one of the partitions has it as null as well? 
   I think it's just the former



##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/BaseAppenderatorDriver.java:
##########
@@ -652,19 +656,16 @@ ListenableFuture<SegmentsAndCommitMetadata> publishInBackground(
                   segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
                 }
               } else {
+                log.errorSegments(ourSegments, "Failed to publish segments");
+                if (publishResult.getErrorMsg() != null && publishResult.getErrorMsg().contains(("Failed to update the metadata Store. The new start metadata is ahead of last commited end state."))) {
+                  throw new ISE(publishResult.getErrorMsg());
+                }
                 // Our segments aren't active. Publish failed for some reason. Clean them up and then throw an error.
                 segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);

Review Comment:
   Yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1371160349


##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -2054,18 +2054,35 @@ protected DataStoreMetadataUpdateResult updateDataSourceMetadataWithHandle(
     }
 
     final boolean startMetadataMatchesExisting;
+    final int startMetadataGreaterThanExisting;
 
     if (oldCommitMetadataFromDb == null) {
       startMetadataMatchesExisting = startMetadata.isValidStart();
+      startMetadataGreaterThanExisting = 1;
     } else {
       // Checking against the last committed metadata.
+      // If the new start sequence number is greater than the end sequence number of last commit compareTo() function will return 1,
+      // 0 in all other cases. It might be because  multiple tasks are publishing the sequence at around same time.
+      startMetadataGreaterThanExisting = startMetadata.asStartMetadata().compareTo(oldCommitMetadataFromDb.asStartMetadata());

Review Comment:
   You could add a condition here to check if metadata is comparable. 



##########
server/src/main/java/org/apache/druid/indexing/overlord/DataSourceMetadata.java:
##########
@@ -36,7 +36,7 @@
 @JsonSubTypes(value = {
     @JsonSubTypes.Type(name = "object", value = ObjectMetadata.class)
 })
-public interface DataSourceMetadata
+public interface DataSourceMetadata extends Comparable<DataSourceMetadata>

Review Comment:
   This interface need not extend Comparable as discussed earlier. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1445979916


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,38 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other, Comparator<SequenceOffsetType> comparator)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      boolean res = false;
+      for (Map.Entry<PartitionIdType, SequenceOffsetType> entry : partitionSequenceNumberMap.entrySet()) {
+        PartitionIdType partitionId = entry.getKey();
+        SequenceOffsetType sequenceOffset = entry.getValue();
+        if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && comparator.compare(sequenceOffset, otherStart.partitionSequenceNumberMap.get(partitionId)) > 0) {

Review Comment:
   It will be true for both. For First the metadata from DB is null and it checks for validStart(). And for the second it will try to match old and new metadata, using a+b equals b+a. So, it will also be true.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1445621990


##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/BaseAppenderatorDriver.java:
##########
@@ -675,12 +676,19 @@ ListenableFuture<SegmentsAndCommitMetadata> publishInBackground(
                 segmentsAndCommitMetadata.getSegments(),
                 "Failed publish, not removing segments"
             );
-            Throwables.propagateIfPossible(e);
-            throw new RuntimeException(e);
+            if (e.getMessage() != null && e.getMessage().contains("Failed to update the metadata Store. The new start metadata is ahead of last commited end state.")) {
+              // we can recover from this error.
+              throw new ExecutionException(e);

Review Comment:
   Why do we need to throw an ExecutionException instead of checking the message itself below?



##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/BaseAppenderatorDriver.java:
##########
@@ -652,19 +656,16 @@ ListenableFuture<SegmentsAndCommitMetadata> publishInBackground(
                   segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
                 }
               } else {
+                log.errorSegments(ourSegments, "Failed to publish segments");
+                if (publishResult.getErrorMsg() != null && publishResult.getErrorMsg().contains(("Failed to update the metadata Store. The new start metadata is ahead of last commited end state."))) {
+                  throw new ISE(publishResult.getErrorMsg());
+                }
                 // Our segments aren't active. Publish failed for some reason. Clean them up and then throw an error.
                 segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);

Review Comment:
   Segments should be cleaned up if the retries are exhausted or an unrecoverable exception is thrown.



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,38 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other, Comparator<SequenceOffsetType> comparator)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      boolean res = false;
+      for (Map.Entry<PartitionIdType, SequenceOffsetType> entry : partitionSequenceNumberMap.entrySet()) {
+        PartitionIdType partitionId = entry.getKey();
+        SequenceOffsetType sequenceOffset = entry.getValue();
+        if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && comparator.compare(sequenceOffset, otherStart.partitionSequenceNumberMap.get(partitionId)) > 0) {

Review Comment:
   If otherStart for partitionId is null and sequenceOffset is not null, should the comparator return 1?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1445644141


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,38 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other, Comparator<SequenceOffsetType> comparator)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      boolean res = false;
+      for (Map.Entry<PartitionIdType, SequenceOffsetType> entry : partitionSequenceNumberMap.entrySet()) {
+        PartitionIdType partitionId = entry.getKey();
+        SequenceOffsetType sequenceOffset = entry.getValue();
+        if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && comparator.compare(sequenceOffset, otherStart.partitionSequenceNumberMap.get(partitionId)) > 0) {

Review Comment:
   @AmatyaAvadhanula, I think this will happen, when the partition reporting offset for the first time, right? So, for this case, there will be no need to retry, As startMetadataMatchesExisting will be true. And it will update the datastore.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1444306068


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -61,4 +62,11 @@ SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
   SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> minus(
       SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other
   );
+
+  /**
+   * Compare this and the other sequence offsets using comparator.
+   * Returns 1, if this sequence is ahead of the other.

Review Comment:
   Hey @AmatyaAvadhanula, as per our discussion I have added a check to verify task partitions are contained within the set in the metadata total set.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1358025616


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   or you can have just KafkaDataSourceMetadata implement Comparable. Then you would check if the metadata is of type comparable and call compare method. Then you don't need `isComparable` method and the interface doesn't change either. That's actually better and easy to change in the future. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1884294746

   @kfaraz - I merged this without looking at your comment. But feel free to comment and @Pankaj260100 can hopefully address those in a follow-up PR. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1447638174


##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -2054,18 +2054,37 @@ protected DataStoreMetadataUpdateResult updateDataSourceMetadataWithHandle(
     }
 
     final boolean startMetadataMatchesExisting;
+    int startMetadataGreaterThanExisting = 0;

Review Comment:
   Yes, we can have a boolean. I have implemented the Comparable for compareTo() method and it returns int. so, I didn't change it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1879734817

   @Pankaj260100 - did you verify this patch in any test cluster? Does it fix the problem? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1367874829


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   The class `OrderedSequenceNumber` may already have the implementation to compare Kafka and Kinesis sequence numbers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1447641302


##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -2054,18 +2054,37 @@ protected DataStoreMetadataUpdateResult updateDataSourceMetadataWithHandle(
     }
 
     final boolean startMetadataMatchesExisting;
+    int startMetadataGreaterThanExisting = 0;

Review Comment:
   `comareTo()` function returns +1, -1 and 0 for greaterThan, lessThan and Equal repectively.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1885194055

   Thanks, @abhishekagarwal87, @AmatyaAvadhanula & @xvrl  for the help. 
   
   >But feel free to comment and @Pankaj260100 can hopefully address those in a follow-up PR.
   
   Sure, I will do that.
      


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1443794109


##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/BaseAppenderatorDriver.java:
##########
@@ -675,12 +676,19 @@
                 segmentsAndCommitMetadata.getSegments(),
                 "Failed publish, not removing segments"
             );
-            Throwables.propagateIfPossible(e);
-            throw new RuntimeException(e);
+            if (e.getMessage() != null && e.getMessage().contains("Failed to update the metadata Store. The new start metadata is ahead of last commited end state.")) {
+              // we can recover from this error.
+              throw new ExecutionException(e);
+            } else {
+              Throwables.propagateIfPossible(e);

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [Throwables.propagateIfPossible](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/6416)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1373516274


##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -2054,18 +2054,35 @@ protected DataStoreMetadataUpdateResult updateDataSourceMetadataWithHandle(
     }
 
     final boolean startMetadataMatchesExisting;
+    final int startMetadataGreaterThanExisting;
 
     if (oldCommitMetadataFromDb == null) {
       startMetadataMatchesExisting = startMetadata.isValidStart();
+      startMetadataGreaterThanExisting = 1;
     } else {
       // Checking against the last committed metadata.
+      // If the new start sequence number is greater than the end sequence number of last commit compareTo() function will return 1,
+      // 0 in all other cases. It might be because  multiple tasks are publishing the sequence at around same time.
+      startMetadataGreaterThanExisting = startMetadata.asStartMetadata().compareTo(oldCommitMetadataFromDb.asStartMetadata());

Review Comment:
   More like 
   ```suggestion
        if (startMetadata instanceof Comparable)
         startMetadataGreaterThanExisting = startMetadata.asStartMetadata().compareTo(oldCommitMetadataFromDb.asStartMetadata());
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1382818853


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -61,4 +62,11 @@ SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
   SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> minus(
       SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other
   );
+
+  /**
+   * Compare this and the other sequence offsets using comparator.
+   * Returns 1, if this sequence is ahead of the other.

Review Comment:
   Isn't it possible that when there are 10 partitions, 6 have strictly greater sequence numbers while the remaining 4 have equal sequence numbers because no new data was added to them?
   I think we should retry in this case as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1363475906


##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaDataSourceMetadata.java:
##########
@@ -58,4 +59,19 @@ protected SeekableStreamDataSourceMetadata<KafkaTopicPartition, Long> createConc
   {
     return new KafkaDataSourceMetadata(seekableStreamSequenceNumbers);
   }
+
+  @Override
+  public int compareTo(DataSourceMetadata other)
+  {
+    if (!getClass().equals(other.getClass())) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+    final SeekableStreamDataSourceMetadata<KafkaTopicPartition, Long> that = (SeekableStreamDataSourceMetadata<KafkaTopicPartition, Long>) other;
+
+    return getSeekableStreamSequenceNumbers().compareTo(that.getSeekableStreamSequenceNumbers());

Review Comment:
   ```suggestion
       return getSeekableStreamSequenceNumbers().compareTo(that.getSeekableStreamSequenceNumbers(), LongComparator());
   ```



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,37 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)

Review Comment:
   ```suggestion
     public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other, Comparator<SequenceOffsetType> comparator)
   ```



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,37 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && (Long) sequenceOffset > (Long) otherStart.partitionSequenceNumberMap.get(partitionId)) {

Review Comment:
   ```suggestion
               if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && comparator.compare(sequenceOffset, otherStart.partitionSequenceNumberMap.get(partitionId) ) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1375631275


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   @Pankaj260100, Don't `KafkaSequenceNumber` and `KinesisSequenceNumber` which extend this class handle the comparison at an individual sequence number's level?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1375634238


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +149,37 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other, Comparator<SequenceOffsetType> comparator)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset

Review Comment:
   Could you also please add a check to compare the partitions to account for repartitioning?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1383357174


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -61,4 +62,11 @@ SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
   SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> minus(
       SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other
   );
+
+  /**
+   * Compare this and the other sequence offsets using comparator.
+   * Returns 1, if this sequence is ahead of the other.

Review Comment:
   I was wondering if it's possible that one partition is strictly greater but the other is strictly lower



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1364958249


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   It could, yes. The solution here is to either retry no matter how current and new offsets compare. Or we could add the comparison to Kinesis too - https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SequenceNumberRange.html - The sequence numbers seem to be numbers in string form but can contain up to 128 digits. It should be possible to compare them without converting them to a number. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1446853736


##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/BaseAppenderatorDriver.java:
##########
@@ -596,91 +597,104 @@
                                   ? null
                                   : ((AppenderatorDriverMetadata) metadata).getCallerMetadata();
     return executor.submit(
-        () -> {
-          try {
-            final ImmutableSet<DataSegment> ourSegments = ImmutableSet.copyOf(pushedAndTombstones);
-            final SegmentPublishResult publishResult = publisher.publishSegments(
-                segmentsToBeOverwritten,
-                ourSegments,
-                outputSegmentsAnnotateFunction,
-                callerMetadata
-            );
-
-            if (publishResult.isSuccess()) {
-              log.info(
-                  "Published [%s] segments with commit metadata [%s]",
-                  segmentsAndCommitMetadata.getSegments().size(),
-                  callerMetadata
-              );
-              log.infoSegments(segmentsAndCommitMetadata.getSegments(), "Published segments");
-            } else {
-              // Publishing didn't affirmatively succeed. However, segments with our identifiers may still be active
-              // now after all, for two possible reasons:
-              //
-              // 1) A replica may have beat us to publishing these segments. In this case we want to delete the
-              //    segments we pushed (if they had unique paths) to avoid wasting space on deep storage.
-              // 2) We may have actually succeeded, but not realized it due to missing the confirmation response
-              //    from the overlord. In this case we do not want to delete the segments we pushed, since they are
-              //    now live!
-
-              final Set<SegmentIdWithShardSpec> segmentsIdentifiers = segmentsAndCommitMetadata
-                  .getSegments()
-                  .stream()
-                  .map(SegmentIdWithShardSpec::fromDataSegment)
-                  .collect(Collectors.toSet());
-
-              final Set<DataSegment> activeSegments = usedSegmentChecker.findUsedSegments(segmentsIdentifiers);
-
-              if (activeSegments.equals(ourSegments)) {
-                log.info(
-                    "Could not publish [%s] segments, but checked and found them already published; continuing.",
-                    ourSegments.size()
-                );
-                log.infoSegments(
-                    segmentsAndCommitMetadata.getSegments(),
-                    "Could not publish segments"
+      () -> {
+        try {
+          RetryUtils.retry(
+              () -> {
+              try {
+                final ImmutableSet<DataSegment> ourSegments = ImmutableSet.copyOf(pushedAndTombstones);
+                final SegmentPublishResult publishResult = publisher.publishSegments(
+                    segmentsToBeOverwritten,
+                    ourSegments,
+                    outputSegmentsAnnotateFunction,
+                    callerMetadata
                 );
 
-                // Clean up pushed segments if they are physically disjoint from the published ones (this means
-                // they were probably pushed by a replica, and with the unique paths option).
-                final boolean physicallyDisjoint = Sets.intersection(
-                    activeSegments.stream().map(DataSegment::getLoadSpec).collect(Collectors.toSet()),
-                    ourSegments.stream().map(DataSegment::getLoadSpec).collect(Collectors.toSet())
-                ).isEmpty();
-
-                if (physicallyDisjoint) {
-                  segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
-                }
-              } else {
-                // Our segments aren't active. Publish failed for some reason. Clean them up and then throw an error.
-                segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
-
-                if (publishResult.getErrorMsg() != null) {
-                  log.errorSegments(ourSegments, "Failed to publish segments");
-                  throw new ISE(
-                      "Failed to publish segments because of [%s]",
-                      publishResult.getErrorMsg()
+                if (publishResult.isSuccess()) {
+                  log.info(
+                      "Published [%s] segments with commit metadata [%s]",
+                      segmentsAndCommitMetadata.getSegments().size(),
+                      callerMetadata
                   );
+                  log.infoSegments(segmentsAndCommitMetadata.getSegments(), "Published segments");
                 } else {
-                  log.errorSegments(ourSegments, "Failed to publish segments");
-                  throw new ISE("Failed to publish segments");
+                  // Publishing didn't affirmatively succeed. However, segments with our identifiers may still be active
+                  // now after all, for two possible reasons:
+                  //
+                  // 1) A replica may have beat us to publishing these segments. In this case we want to delete the
+                  //    segments we pushed (if they had unique paths) to avoid wasting space on deep storage.
+                  // 2) We may have actually succeeded, but not realized it due to missing the confirmation response
+                  //    from the overlord. In this case we do not want to delete the segments we pushed, since they are
+                  //    now live!
+
+                  final Set<SegmentIdWithShardSpec> segmentsIdentifiers = segmentsAndCommitMetadata
+                      .getSegments()
+                      .stream()
+                      .map(SegmentIdWithShardSpec::fromDataSegment)
+                      .collect(Collectors.toSet());
+
+                  final Set<DataSegment> activeSegments = usedSegmentChecker.findUsedSegments(segmentsIdentifiers);
+
+                  if (activeSegments.equals(ourSegments)) {
+                    log.info(
+                        "Could not publish [%s] segments, but checked and found them already published; continuing.",
+                        ourSegments.size()
+                    );
+                    log.infoSegments(
+                        segmentsAndCommitMetadata.getSegments(),
+                        "Could not publish segments"
+                    );
+
+                    // Clean up pushed segments if they are physically disjoint from the published ones (this means
+                    // they were probably pushed by a replica, and with the unique paths option).
+                    final boolean physicallyDisjoint = Sets.intersection(
+                        activeSegments.stream().map(DataSegment::getLoadSpec).collect(Collectors.toSet()),
+                        ourSegments.stream().map(DataSegment::getLoadSpec).collect(Collectors.toSet())
+                    ).isEmpty();
+
+                    if (physicallyDisjoint) {
+                      segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
+                    }
+                  } else {
+                    log.errorSegments(ourSegments, "Failed to publish segments");
+                    if (publishResult.getErrorMsg() != null && publishResult.getErrorMsg().contains(("Failed to update the metadata Store. The new start metadata is ahead of last commited end state."))) {
+                      throw new ISE(publishResult.getErrorMsg());
+                    }
+                    // Our segments aren't active. Publish failed for some reason. Clean them up and then throw an error.
+                    segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
+                    if (publishResult.getErrorMsg() != null) {
+                      throw new ISE("Failed to publish segments because of [%s]", publishResult.getErrorMsg());
+                    }
+                    throw new ISE("Failed to publish segments");
+                  }
                 }
               }
-            }
-          }
-          catch (Exception e) {
-            // Must not remove segments here, we aren't sure if our transaction succeeded or not.
-            log.noStackTrace().warn(e, "Failed publish");
-            log.warnSegments(
-                segmentsAndCommitMetadata.getSegments(),
-                "Failed publish, not removing segments"
-            );
-            Throwables.propagateIfPossible(e);
-            throw new RuntimeException(e);
+              catch (Exception e) {
+                // Must not remove segments here, we aren't sure if our transaction succeeded or not.
+                log.noStackTrace().warn(e, "Failed publish");
+                log.warnSegments(
+                    segmentsAndCommitMetadata.getSegments(),
+                    "Failed publish, not removing segments"
+                );
+                Throwables.propagateIfPossible(e);
+                throw new RuntimeException(e);
+              }
+              return segmentsAndCommitMetadata;
+            },
+              e -> (e.getMessage() != null && e.getMessage().contains("Failed to update the metadata Store. The new start metadata is ahead of last commited end state.")),
+              RetryUtils.DEFAULT_MAX_TRIES
+          );
+        }
+        catch (Exception e) {
+          if (e.getMessage() != null && e.getMessage().contains("Failed to update the metadata Store. The new start metadata is ahead of last commited end state.")) {
+            // Publish failed for some reason. Clean them up and then throw an error.
+            segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
           }
-
-          return segmentsAndCommitMetadata;
+          Throwables.propagateIfPossible(e);

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [Throwables.propagateIfPossible](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/6435)



##########
server/src/main/java/org/apache/druid/segment/realtime/appenderator/BaseAppenderatorDriver.java:
##########
@@ -596,91 +597,104 @@
                                   ? null
                                   : ((AppenderatorDriverMetadata) metadata).getCallerMetadata();
     return executor.submit(
-        () -> {
-          try {
-            final ImmutableSet<DataSegment> ourSegments = ImmutableSet.copyOf(pushedAndTombstones);
-            final SegmentPublishResult publishResult = publisher.publishSegments(
-                segmentsToBeOverwritten,
-                ourSegments,
-                outputSegmentsAnnotateFunction,
-                callerMetadata
-            );
-
-            if (publishResult.isSuccess()) {
-              log.info(
-                  "Published [%s] segments with commit metadata [%s]",
-                  segmentsAndCommitMetadata.getSegments().size(),
-                  callerMetadata
-              );
-              log.infoSegments(segmentsAndCommitMetadata.getSegments(), "Published segments");
-            } else {
-              // Publishing didn't affirmatively succeed. However, segments with our identifiers may still be active
-              // now after all, for two possible reasons:
-              //
-              // 1) A replica may have beat us to publishing these segments. In this case we want to delete the
-              //    segments we pushed (if they had unique paths) to avoid wasting space on deep storage.
-              // 2) We may have actually succeeded, but not realized it due to missing the confirmation response
-              //    from the overlord. In this case we do not want to delete the segments we pushed, since they are
-              //    now live!
-
-              final Set<SegmentIdWithShardSpec> segmentsIdentifiers = segmentsAndCommitMetadata
-                  .getSegments()
-                  .stream()
-                  .map(SegmentIdWithShardSpec::fromDataSegment)
-                  .collect(Collectors.toSet());
-
-              final Set<DataSegment> activeSegments = usedSegmentChecker.findUsedSegments(segmentsIdentifiers);
-
-              if (activeSegments.equals(ourSegments)) {
-                log.info(
-                    "Could not publish [%s] segments, but checked and found them already published; continuing.",
-                    ourSegments.size()
-                );
-                log.infoSegments(
-                    segmentsAndCommitMetadata.getSegments(),
-                    "Could not publish segments"
+      () -> {
+        try {
+          RetryUtils.retry(
+              () -> {
+              try {
+                final ImmutableSet<DataSegment> ourSegments = ImmutableSet.copyOf(pushedAndTombstones);
+                final SegmentPublishResult publishResult = publisher.publishSegments(
+                    segmentsToBeOverwritten,
+                    ourSegments,
+                    outputSegmentsAnnotateFunction,
+                    callerMetadata
                 );
 
-                // Clean up pushed segments if they are physically disjoint from the published ones (this means
-                // they were probably pushed by a replica, and with the unique paths option).
-                final boolean physicallyDisjoint = Sets.intersection(
-                    activeSegments.stream().map(DataSegment::getLoadSpec).collect(Collectors.toSet()),
-                    ourSegments.stream().map(DataSegment::getLoadSpec).collect(Collectors.toSet())
-                ).isEmpty();
-
-                if (physicallyDisjoint) {
-                  segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
-                }
-              } else {
-                // Our segments aren't active. Publish failed for some reason. Clean them up and then throw an error.
-                segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
-
-                if (publishResult.getErrorMsg() != null) {
-                  log.errorSegments(ourSegments, "Failed to publish segments");
-                  throw new ISE(
-                      "Failed to publish segments because of [%s]",
-                      publishResult.getErrorMsg()
+                if (publishResult.isSuccess()) {
+                  log.info(
+                      "Published [%s] segments with commit metadata [%s]",
+                      segmentsAndCommitMetadata.getSegments().size(),
+                      callerMetadata
                   );
+                  log.infoSegments(segmentsAndCommitMetadata.getSegments(), "Published segments");
                 } else {
-                  log.errorSegments(ourSegments, "Failed to publish segments");
-                  throw new ISE("Failed to publish segments");
+                  // Publishing didn't affirmatively succeed. However, segments with our identifiers may still be active
+                  // now after all, for two possible reasons:
+                  //
+                  // 1) A replica may have beat us to publishing these segments. In this case we want to delete the
+                  //    segments we pushed (if they had unique paths) to avoid wasting space on deep storage.
+                  // 2) We may have actually succeeded, but not realized it due to missing the confirmation response
+                  //    from the overlord. In this case we do not want to delete the segments we pushed, since they are
+                  //    now live!
+
+                  final Set<SegmentIdWithShardSpec> segmentsIdentifiers = segmentsAndCommitMetadata
+                      .getSegments()
+                      .stream()
+                      .map(SegmentIdWithShardSpec::fromDataSegment)
+                      .collect(Collectors.toSet());
+
+                  final Set<DataSegment> activeSegments = usedSegmentChecker.findUsedSegments(segmentsIdentifiers);
+
+                  if (activeSegments.equals(ourSegments)) {
+                    log.info(
+                        "Could not publish [%s] segments, but checked and found them already published; continuing.",
+                        ourSegments.size()
+                    );
+                    log.infoSegments(
+                        segmentsAndCommitMetadata.getSegments(),
+                        "Could not publish segments"
+                    );
+
+                    // Clean up pushed segments if they are physically disjoint from the published ones (this means
+                    // they were probably pushed by a replica, and with the unique paths option).
+                    final boolean physicallyDisjoint = Sets.intersection(
+                        activeSegments.stream().map(DataSegment::getLoadSpec).collect(Collectors.toSet()),
+                        ourSegments.stream().map(DataSegment::getLoadSpec).collect(Collectors.toSet())
+                    ).isEmpty();
+
+                    if (physicallyDisjoint) {
+                      segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
+                    }
+                  } else {
+                    log.errorSegments(ourSegments, "Failed to publish segments");
+                    if (publishResult.getErrorMsg() != null && publishResult.getErrorMsg().contains(("Failed to update the metadata Store. The new start metadata is ahead of last commited end state."))) {
+                      throw new ISE(publishResult.getErrorMsg());
+                    }
+                    // Our segments aren't active. Publish failed for some reason. Clean them up and then throw an error.
+                    segmentsAndCommitMetadata.getSegments().forEach(dataSegmentKiller::killQuietly);
+                    if (publishResult.getErrorMsg() != null) {
+                      throw new ISE("Failed to publish segments because of [%s]", publishResult.getErrorMsg());
+                    }
+                    throw new ISE("Failed to publish segments");
+                  }
                 }
               }
-            }
-          }
-          catch (Exception e) {
-            // Must not remove segments here, we aren't sure if our transaction succeeded or not.
-            log.noStackTrace().warn(e, "Failed publish");
-            log.warnSegments(
-                segmentsAndCommitMetadata.getSegments(),
-                "Failed publish, not removing segments"
-            );
-            Throwables.propagateIfPossible(e);
-            throw new RuntimeException(e);
+              catch (Exception e) {
+                // Must not remove segments here, we aren't sure if our transaction succeeded or not.
+                log.noStackTrace().warn(e, "Failed publish");
+                log.warnSegments(
+                    segmentsAndCommitMetadata.getSegments(),
+                    "Failed publish, not removing segments"
+                );
+                Throwables.propagateIfPossible(e);

Review Comment:
   ## Deprecated method or constructor invocation
   
   Invoking [Throwables.propagateIfPossible](1) should be avoided because it has been deprecated.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/6434)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1446956883


##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -2054,18 +2054,37 @@ protected DataStoreMetadataUpdateResult updateDataSourceMetadataWithHandle(
     }
 
     final boolean startMetadataMatchesExisting;
+    int startMetadataGreaterThanExisting = 0;

Review Comment:
   Can this be a boolean instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1853629473

   It appears that retries on the Overlord happen within a transaction and that would explain the observation.
   Perhaps the OverlordResource could return a status code other than 400 for certain exceptions, and the task could retry submitting the task action in such cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1383004203


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -61,4 +62,11 @@ SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
   SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> minus(
       SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other
   );
+
+  /**
+   * Compare this and the other sequence offsets using comparator.
+   * Returns 1, if this sequence is ahead of the other.

Review Comment:
   @AmatyaAvadhanula, This case is covered. We will retry when atleast one partition sequence number is strictly greater than the other. We will compare all partition offsets, and if we find one of the partition offsets greater than the other offset, we update the res variable as true. And retry in that case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "github-code-scanning[bot] (via GitHub)" <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1356679692


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   ## Missing catch of NumberFormatException
   
   Potential uncaught 'java.lang.NumberFormatException'.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/5895)



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamStartSequenceNumbers.java:
##########
@@ -161,6 +162,40 @@
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamStartSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamStartSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   ## Missing catch of NumberFormatException
   
   Potential uncaught 'java.lang.NumberFormatException'.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/5898)



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamStartSequenceNumbers.java:
##########
@@ -161,6 +162,40 @@
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamStartSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamStartSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   ## Missing catch of NumberFormatException
   
   Potential uncaught 'java.lang.NumberFormatException'.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/5897)



##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   ## Missing catch of NumberFormatException
   
   Potential uncaught 'java.lang.NumberFormatException'.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/5896)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1373252387


##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -2054,18 +2054,35 @@ protected DataStoreMetadataUpdateResult updateDataSourceMetadataWithHandle(
     }
 
     final boolean startMetadataMatchesExisting;
+    final int startMetadataGreaterThanExisting;
 
     if (oldCommitMetadataFromDb == null) {
       startMetadataMatchesExisting = startMetadata.isValidStart();
+      startMetadataGreaterThanExisting = 1;
     } else {
       // Checking against the last committed metadata.
+      // If the new start sequence number is greater than the end sequence number of last commit compareTo() function will return 1,
+      // 0 in all other cases. It might be because  multiple tasks are publishing the sequence at around same time.
+      startMetadataGreaterThanExisting = startMetadata.asStartMetadata().compareTo(oldCommitMetadataFromDb.asStartMetadata());

Review Comment:
   Somethings like this?
   ```suggestion
       if (startMetadata instanceof KafkaDataSourceMetadata)
         startMetadataGreaterThanExisting = startMetadata.asStartMetadata().compareTo(oldCommitMetadataFromDb.asStartMetadata());
   ```
   



##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -2054,18 +2054,35 @@ protected DataStoreMetadataUpdateResult updateDataSourceMetadataWithHandle(
     }
 
     final boolean startMetadataMatchesExisting;
+    final int startMetadataGreaterThanExisting;
 
     if (oldCommitMetadataFromDb == null) {
       startMetadataMatchesExisting = startMetadata.isValidStart();
+      startMetadataGreaterThanExisting = 1;
     } else {
       // Checking against the last committed metadata.
+      // If the new start sequence number is greater than the end sequence number of last commit compareTo() function will return 1,
+      // 0 in all other cases. It might be because  multiple tasks are publishing the sequence at around same time.
+      startMetadataGreaterThanExisting = startMetadata.asStartMetadata().compareTo(oldCommitMetadataFromDb.asStartMetadata());

Review Comment:
   Like this?
   ```suggestion
       if (startMetadata instanceof KafkaDataSourceMetadata)
         startMetadataGreaterThanExisting = startMetadata.asStartMetadata().compareTo(oldCommitMetadataFromDb.asStartMetadata());
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1373248898


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   No, OrderedSequenceNumber does not have implementation to compare. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1373726645


##########
server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java:
##########
@@ -2054,18 +2054,35 @@ protected DataStoreMetadataUpdateResult updateDataSourceMetadataWithHandle(
     }
 
     final boolean startMetadataMatchesExisting;
+    final int startMetadataGreaterThanExisting;
 
     if (oldCommitMetadataFromDb == null) {
       startMetadataMatchesExisting = startMetadata.isValidStart();
+      startMetadataGreaterThanExisting = 1;
     } else {
       // Checking against the last committed metadata.
+      // If the new start sequence number is greater than the end sequence number of last commit compareTo() function will return 1,
+      // 0 in all other cases. It might be because  multiple tasks are publishing the sequence at around same time.
+      startMetadataGreaterThanExisting = startMetadata.asStartMetadata().compareTo(oldCommitMetadataFromDb.asStartMetadata());

Review Comment:
   Oh okay. Now, I understand. Thanks. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "github-advanced-security[bot] (via GitHub)" <gi...@apache.org>.
github-advanced-security[bot] commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1374640533


##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaDataSourceMetadata.java:
##########
@@ -26,9 +26,11 @@
 import org.apache.druid.indexing.seekablestream.SeekableStreamDataSourceMetadata;
 import org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers;
 import org.apache.druid.indexing.seekablestream.SeekableStreamSequenceNumbers;
+import org.apache.druid.java.util.common.IAE;
 
-public class KafkaDataSourceMetadata extends SeekableStreamDataSourceMetadata<KafkaTopicPartition, Long>
-{
+import java.util.Comparator;
+
+public class KafkaDataSourceMetadata extends SeekableStreamDataSourceMetadata<KafkaTopicPartition, Long> implements Comparable<KafkaDataSourceMetadata> {

Review Comment:
   ## Inconsistent compareTo
   
   This class declares [compareTo](1) but inherits equals; the two could be inconsistent.
   
   [Show more details](https://github.com/apache/druid/security/code-scanning/5937)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "xvrl (via GitHub)" <gi...@apache.org>.
xvrl commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1357441720


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   I'm not sure we can assume sequenceOffsets can always be parsed as long. This seems to be a fairly broad assumption. Someone more familiar with the Kinesis supervisor should chime in.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1446923072


##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaDataSourceMetadata.java:
##########
@@ -58,4 +60,17 @@ protected SeekableStreamDataSourceMetadata<KafkaTopicPartition, Long> createConc
   {
     return new KafkaDataSourceMetadata(seekableStreamSequenceNumbers);
   }
+
+  @Override
+  public int compareTo(KafkaDataSourceMetadata other)

Review Comment:
   Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1884246691

   Thank you @Pankaj260100. LGTM!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1375643913


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -61,4 +62,11 @@ SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
   SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> minus(
       SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other
   );
+
+  /**
+   * Compare this and the other sequence offsets using comparator.
+   * Returns 1, if this sequence is ahead of the other.

Review Comment:
   Could you please elaborate that this sequence being ahead of the other means that the partition-wise sequence number is greater than or equal to the other's with a strict greater than condition for at least one of the partitions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1784446163

   @Pankaj260100, since https://github.com/apache/druid/issues/15054 is reproducible, could you please test this patch on a Druid cluster and indicate the same in the PR description as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1360538580


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +148,40 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && Long.parseLong(String.valueOf(sequenceOffset)) > Long.parseLong(String.valueOf(otherStart.partitionSequenceNumberMap.get(partitionId)))) {

Review Comment:
   @abhishekagarwal87, I have implemented comparable method only in kafkaDataSourceMetadata. And kinesisDataSourceMetadata will return 0(default).  PTAL



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "xvrl (via GitHub)" <gi...@apache.org>.
xvrl commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1357444512


##########
extensions-contrib/materialized-view-maintenance/src/main/java/org/apache/druid/indexing/materializedview/DerivativeDataSourceMetadata.java:
##########
@@ -85,6 +85,12 @@ public boolean matches(DataSourceMetadata other)
     return equals(other);
   }
 
+  @Override
+  public int compareTo(DataSourceMetadata o)
+  {
+    return -1;

Review Comment:
   always returning `-1` wouldn't satisfy the Comparable contract here. it would mean A < B and B < A for any two objects A and B implementing this interface. We might need to explain better what the contract implies or throw an exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "xvrl (via GitHub)" <gi...@apache.org>.
xvrl commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1357442276


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamSequenceNumbers.java:
##########
@@ -32,7 +32,7 @@
     @Type(name = "start", value = SeekableStreamStartSequenceNumbers.class),
     @Type(name = "end", value = SeekableStreamEndSequenceNumbers.class)
 })
-public interface SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType>
+public interface SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> extends Comparable<SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType>>

Review Comment:
   can we add some Javadocs to explain what Comparable means in this context and how it applies to sequence offsets?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1815840339

   @Pankaj260100 - can you elaborate a bit further? what is the 1st task and what is the 2nd task? can you use the same terms as you used in the issue you filed (#15054)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1814764084

    @AmatyaAvadhanula @abhishekagarwal87 @xvrl , I tried to test this patch in druid 25 in one of dev druid cluster and whenever a task failed to update the metadata store, it start retrying and then the retry logic execute in sequence like first it complete 10 retries for 1 task then for other task and then ingestion lag start increasing, I was expecting it to happen in parallel. Any idea why this is happening?
    For ex:
    `Retry for 1st task 
    starting from: 13:34:23.173 UTC
    ends at: 13:38:40.378 UTC` 
    `Retry for 2nd task
    starts at: 13:38:40.423 UTC
   ends at: 13:38:59.965 UTC
    `


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #15141:
URL: https://github.com/apache/druid/pull/15141#discussion_r1375420281


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamEndSequenceNumbers.java:
##########
@@ -147,6 +149,37 @@ public SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> plus(
     }
   }
 
+  @Override
+  public int compareTo(SeekableStreamSequenceNumbers<PartitionIdType, SequenceOffsetType> other, Comparator<SequenceOffsetType> comparator)
+  {
+    if (this.getClass() != other.getClass()) {
+      throw new IAE(
+          "Expected instance of %s, got %s",
+          this.getClass().getName(),
+          other.getClass().getName()
+      );
+    }
+
+    final SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType> otherStart =
+        (SeekableStreamEndSequenceNumbers<PartitionIdType, SequenceOffsetType>) other;
+
+    if (stream.equals(otherStart.stream)) {
+      //Same stream, compare the offset
+      AtomicReference<Boolean> res = new AtomicReference<>(false);
+      partitionSequenceNumberMap.forEach(
+          (partitionId, sequenceOffset) -> {
+            if (otherStart.partitionSequenceNumberMap.get(partitionId) != null && comparator.compare(sequenceOffset, otherStart.partitionSequenceNumberMap.get(partitionId)) > 0) {
+              res.set(true);
+            }
+          }
+      );

Review Comment:
   Nit - You don't need AtomicReference here. could use a simple for each loop for map.entryset



##########
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaDataSourceMetadata.java:
##########
@@ -58,4 +60,17 @@ protected SeekableStreamDataSourceMetadata<KafkaTopicPartition, Long> createConc
   {
     return new KafkaDataSourceMetadata(seekableStreamSequenceNumbers);
   }
+
+  @Override
+  public int compareTo(KafkaDataSourceMetadata other)

Review Comment:
   Please add some comments as to how this method is to be used. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Adding retries to update the metadata store instead of failure (druid)

Posted by "Pankaj260100 (via GitHub)" <gi...@apache.org>.
Pankaj260100 commented on PR #15141:
URL: https://github.com/apache/druid/pull/15141#issuecomment-1879735940

   @abhishekagarwal87, Yes I tested this patch on the test druid cluster. I submitted the config twice, So there were 2 sets of tasks publishing simultaneously. From logs, I verified retries happened and there was no task failed.    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org