[GitHub] [hudi] yihua commented on a diff in pull request #7921: [HUDI-5764] Roll back delta commits from `HoodieIndexer` lazily in metadata table

["yihua (via GitHub)" <gi...@apache.org>]

yihua commented on code in PR #7921:
URL: https://github.com/apache/hudi/pull/7921#discussion_r1103472610


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1279,8 +1279,21 @@ protected void tryUpgrade(HoodieTableMetaClient metaClient, Option<String> insta
     }
   }
 
+  /**
+   * Rolls back the failed delta commits corresponding to the indexing action.
+   * <p>
+   * TODO(HUDI-5733): This should be cleaned up once the proper fix of rollbacks
+   *  in the metadata table is landed.
+   *
+   * @return {@code true} if rollback happens; {@code false} otherwise.
+   */
+  public boolean lazyRollbackFailedIndexing() {
+    return tableServiceClient.lazyRollbackFailedIndexing();

Review Comment:
   This API has to be added so that we can reuse part of rollback logic inside `BaseHoodieTableServiceClient`.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -699,6 +740,19 @@ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, H
     Stream<HoodieInstant> inflightInstantsStream = getInflightTimelineExcludeCompactionAndClustering(metaClient)
         .getReverseOrderedInstants();
     if (cleaningPolicy.isEager()) {
+      // Metadata table uses eager cleaning policy, but we need to exclude inflight delta commits
+      // from the async indexer (`HoodieIndexer`).
+      // TODO(HUDI-5733): This should be cleaned up once the proper fix of rollbacks in the
+      //  metadata table is landed.
+      if (HoodieTableMetadata.isMetadataTable(metaClient.getBasePathV2().toString())) {
+        return inflightInstantsStream.map(HoodieInstant::getTimestamp).filter(entry -> {
+          if (curInstantTime.isPresent()) {
+            return !entry.equals(curInstantTime.get());
+          } else {
+            return !isDeltaCommitFromIndexing(entry);

Review Comment:
   This is to exclude the inflight delta commits due to async indexing from the eager rollback.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1069,6 +1073,7 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
     // delta commits synced over will not have an instant time lesser than the last completed instant on the
     // metadata table.
     writeClient.clean(instantTime + "002");
+    writeClient.lazyRollbackFailedIndexing();

Review Comment:
   Lazy rollback of inflight delta commits from async indexing only.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org