You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "danny0405 (via GitHub)" <gi...@apache.org> on 2024/05/02 10:41:03 UTC

[PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

danny0405 opened a new pull request, #11137:
URL: https://github.com/apache/hudi/pull/11137

   ### Change Logs
   
   Streaming ingestion pipeline requires the support because a pending instant is almost always on the timeline.
   
   ### Impact
   
   none
   
   ### Risk level (write none, low medium or high below)
   
   none
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590531114


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -925,6 +925,37 @@ public void testMetadataTableCompactionWithPendingInstants() throws Exception {
     assertEquals(HoodieInstantTimeGenerator.instantTimeMinusMillis(inflightInstant2, 1L), tableMetadata.getLatestCompactionTime().get());
   }
 
+  @Test
+  public void testInitializeMetadataTableWithPendingInstant() throws Exception {
+    init(COPY_ON_WRITE, false);

Review Comment:
   If we are going to test for only one table type, shall we test for MOR instead of COW? If the initialization works for MOR, it should work for COW as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590551681


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -261,12 +260,8 @@ private void bootstrapAndVerifyFailure() throws Exception {
     writeConfig = getWriteConfig(true, true);
     initWriteConfigAndMetatableWriter(writeConfig, true);
     syncTableMetadata(writeConfig);
-    try {
-      validateMetadata(testTable);
-      Assertions.fail("Should have failed");
-    } catch (IllegalStateException e) {
-      // expected
-    }
+    Assertions.assertThrows(AssertionFailedError.class, () -> validateMetadata(testTable),

Review Comment:
   Yeah, then we need to throw a specific expection from the `validateMetadata`. In side `validateMetadata`, we have many assertions for the equality of partitions and file list, just change the class to Java `Error`, and also assert the specific error msg.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2090322926

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590549815


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -463,11 +460,11 @@ private String generateUniqueCommitInstantTime(String initializationTime) {
     if (HoodieTableMetadataUtil.isIndexingCommit(dataIndexTimeline, initializationTime)) {
       return initializationTime;
     }
-    // Add suffix to initializationTime to find an unused instant time for the next index initialization.
+    // otherwise yields the timestamp on the fly.
     // This function would be called multiple times in a single application if multiple indexes are being
     // initialized one after the other.
     for (int offset = 0; ; ++offset) {
-      final String commitInstantTime = HoodieTableMetadataUtil.createIndexInitTimestamp(initializationTime, offset);
+      final String commitInstantTime = HoodieInstantTimeGenerator.instantTimePlusMillis(SOLO_COMMIT_TIMESTAMP, offset);

Review Comment:
   For non-empty table, we also switch to start from the `SOLO_COMMIT_TIMESTAMP`. That's the main change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590550442


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -925,6 +925,37 @@ public void testMetadataTableCompactionWithPendingInstants() throws Exception {
     assertEquals(HoodieInstantTimeGenerator.instantTimeMinusMillis(inflightInstant2, 1L), tableMetadata.getLatestCompactionTime().get());
   }
 
+  @Test
+  public void testInitializeMetadataTableWithPendingInstant() throws Exception {
+    init(COPY_ON_WRITE, false);

Review Comment:
   The table type is not a factor of the functionality, maybe we can just paramatize this test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590669196


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -167,20 +167,19 @@ public void testMetadataBootstrapInflightCommit() throws Exception {
     HoodieTableType tableType = COPY_ON_WRITE;
     init(tableType, false);
 
+    // In real production env, bootstrap action can only happen on empty table,
+    // otherwise we need to roll back the previous bootstrap first,
+    // see 'SparkBootstrapCommitActionExecutor.execute' for more details.
     doPreBootstrapWriteOperation(testTable, INSERT, "0000001");
     doPreBootstrapWriteOperation(testTable, "0000002");
     // add an inflight commit
     HoodieCommitMetadata inflightCommitMeta = testTable.doWriteOperation("00000007", UPSERT, emptyList(),
-        asList("p1", "p2"), 2, true, true);
+        asList("p1", "p2"), 2, false, true);

Review Comment:
   got it. thanks for the clarification.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2095426039

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662",
       "triggerID" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4c6c60d3b1bdde2c780374b44962e77a6120015f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23683",
       "triggerID" : "4c6c60d3b1bdde2c780374b44962e77a6120015f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4c6c60d3b1bdde2c780374b44962e77a6120015f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23683) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590550067


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -925,6 +925,37 @@ public void testMetadataTableCompactionWithPendingInstants() throws Exception {
     assertEquals(HoodieInstantTimeGenerator.instantTimeMinusMillis(inflightInstant2, 1L), tableMetadata.getLatestCompactionTime().get());
   }
 
+  @Test
+  public void testInitializeMetadataTableWithPendingInstant() throws Exception {
+    init(COPY_ON_WRITE, false);
+    initWriteConfigAndMetatableWriter(writeConfig, false);
+    doWriteOperation(testTable, metaClient.createNewInstantTime(), INSERT);
+    doWriteOperation(testTable, metaClient.createNewInstantTime(), INSERT);
+
+    // test multi-writer scenario. let's add 1,2,3,4 where 1,2,4 succeeded, but 3 is still inflight. so latest delta commit in MDT is 4, while 3 is still pending
+    // in DT and not seen by MDT yet. compaction should not trigger until 3 goes to completion.

Review Comment:
   yeah, I can remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2092359191

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a668de4b47df64e2d09b8c1bd0a172271c41a7e3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590551681


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -261,12 +260,8 @@ private void bootstrapAndVerifyFailure() throws Exception {
     writeConfig = getWriteConfig(true, true);
     initWriteConfigAndMetatableWriter(writeConfig, true);
     syncTableMetadata(writeConfig);
-    try {
-      validateMetadata(testTable);
-      Assertions.fail("Should have failed");
-    } catch (IllegalStateException e) {
-      // expected
-    }
+    Assertions.assertThrows(AssertionFailedError.class, () -> validateMetadata(testTable),

Review Comment:
   Yeah, then we need to throw a specific expection from the `validateMetadata`. Just change the class to Java Error, and also assert the specific error msg.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590551681


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -261,12 +260,8 @@ private void bootstrapAndVerifyFailure() throws Exception {
     writeConfig = getWriteConfig(true, true);
     initWriteConfigAndMetatableWriter(writeConfig, true);
     syncTableMetadata(writeConfig);
-    try {
-      validateMetadata(testTable);
-      Assertions.fail("Should have failed");
-    } catch (IllegalStateException e) {
-      // expected
-    }
+    Assertions.assertThrows(AssertionFailedError.class, () -> validateMetadata(testTable),

Review Comment:
   Yeah, then we need to throw a specific expection from the `validateMetadata`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2092096715

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614) 
   * a668de4b47df64e2d09b8c1bd0a172271c41a7e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2094533717

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a668de4b47df64e2d09b8c1bd0a172271c41a7e3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644) 
   * adc1380cb496881fd2f1c8b30aa059759c7c5c9c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2094573804

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662",
       "triggerID" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * adc1380cb496881fd2f1c8b30aa059759c7c5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590529782


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -463,11 +460,11 @@ private String generateUniqueCommitInstantTime(String initializationTime) {
     if (HoodieTableMetadataUtil.isIndexingCommit(dataIndexTimeline, initializationTime)) {
       return initializationTime;
     }
-    // Add suffix to initializationTime to find an unused instant time for the next index initialization.
+    // otherwise yields the timestamp on the fly.
     // This function would be called multiple times in a single application if multiple indexes are being
     // initialized one after the other.
     for (int offset = 0; ; ++offset) {
-      final String commitInstantTime = HoodieTableMetadataUtil.createIndexInitTimestamp(initializationTime, offset);
+      final String commitInstantTime = HoodieInstantTimeGenerator.instantTimePlusMillis(SOLO_COMMIT_TIMESTAMP, offset);

Review Comment:
   Why do we need the SOLO_COMMIT_TIMESTAMP again here? I think the initializationTime passed to this method should already be SOLO_COMMIT_TIMESTAMP if the timeline had no completed instants right? We do this in https://github.com/apache/hudi/blob/d4f55f193cd79f317d4fc021cc741554ce8cf6cd/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java#L250



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -463,11 +460,11 @@ private String generateUniqueCommitInstantTime(String initializationTime) {
     if (HoodieTableMetadataUtil.isIndexingCommit(dataIndexTimeline, initializationTime)) {
       return initializationTime;
     }
-    // Add suffix to initializationTime to find an unused instant time for the next index initialization.
+    // otherwise yields the timestamp on the fly.

Review Comment:
   let's move this comment to line 459, just before if blok.



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -167,20 +167,19 @@ public void testMetadataBootstrapInflightCommit() throws Exception {
     HoodieTableType tableType = COPY_ON_WRITE;
     init(tableType, false);
 
+    // In real production env, bootstrap action can only happen on empty table,
+    // otherwise we need to roll back the previous bootstrap first,
+    // see 'SparkBootstrapCommitActionExecutor.execute' for more details.
     doPreBootstrapWriteOperation(testTable, INSERT, "0000001");
     doPreBootstrapWriteOperation(testTable, "0000002");
     // add an inflight commit
     HoodieCommitMetadata inflightCommitMeta = testTable.doWriteOperation("00000007", UPSERT, emptyList(),
-        asList("p1", "p2"), 2, true, true);
+        asList("p1", "p2"), 2, false, true);

Review Comment:
   setting to `false` won't do any bootstrap. Can you please explain why bootstrap is set to false?



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -925,6 +925,37 @@ public void testMetadataTableCompactionWithPendingInstants() throws Exception {
     assertEquals(HoodieInstantTimeGenerator.instantTimeMinusMillis(inflightInstant2, 1L), tableMetadata.getLatestCompactionTime().get());
   }
 
+  @Test
+  public void testInitializeMetadataTableWithPendingInstant() throws Exception {
+    init(COPY_ON_WRITE, false);
+    initWriteConfigAndMetatableWriter(writeConfig, false);
+    doWriteOperation(testTable, metaClient.createNewInstantTime(), INSERT);
+    doWriteOperation(testTable, metaClient.createNewInstantTime(), INSERT);
+
+    // test multi-writer scenario. let's add 1,2,3,4 where 1,2,4 succeeded, but 3 is still inflight. so latest delta commit in MDT is 4, while 3 is still pending
+    // in DT and not seen by MDT yet. compaction should not trigger until 3 goes to completion.

Review Comment:
   Test doesn't seem to do as commented.



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -925,6 +925,37 @@ public void testMetadataTableCompactionWithPendingInstants() throws Exception {
     assertEquals(HoodieInstantTimeGenerator.instantTimeMinusMillis(inflightInstant2, 1L), tableMetadata.getLatestCompactionTime().get());
   }
 
+  @Test
+  public void testInitializeMetadataTableWithPendingInstant() throws Exception {
+    init(COPY_ON_WRITE, false);

Review Comment:
   If we are going to for only one table type, shall we test for MOR instead of COW? If the initialization works for MOR, it should work for COW as well.



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -261,12 +260,8 @@ private void bootstrapAndVerifyFailure() throws Exception {
     writeConfig = getWriteConfig(true, true);
     initWriteConfigAndMetatableWriter(writeConfig, true);
     syncTableMetadata(writeConfig);
-    try {
-      validateMetadata(testTable);
-      Assertions.fail("Should have failed");
-    } catch (IllegalStateException e) {
-      // expected
-    }
+    Assertions.assertThrows(AssertionFailedError.class, () -> validateMetadata(testTable),

Review Comment:
   Should there be an assert or validation in the non-test code itself? I think asserting `AssertionFailedError` is like swallowing any exception. We should assert some specific exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590551681


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -261,12 +260,8 @@ private void bootstrapAndVerifyFailure() throws Exception {
     writeConfig = getWriteConfig(true, true);
     initWriteConfigAndMetatableWriter(writeConfig, true);
     syncTableMetadata(writeConfig);
-    try {
-      validateMetadata(testTable);
-      Assertions.fail("Should have failed");
-    } catch (IllegalStateException e) {
-      // expected
-    }
+    Assertions.assertThrows(AssertionFailedError.class, () -> validateMetadata(testTable),

Review Comment:
   Yeah, then we need to throw a specific expection from the `validateMetadata`. In side `validateMetadata`, we have many assertions for the equality of partitions and file list, and here we want to catch the assertion error, so just change the class to Java `Error`, and also assert the specific error msg.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2095271629

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662",
       "triggerID" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4c6c60d3b1bdde2c780374b44962e77a6120015f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4c6c60d3b1bdde2c780374b44962e77a6120015f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * adc1380cb496881fd2f1c8b30aa059759c7c5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662) 
   * 4c6c60d3b1bdde2c780374b44962e77a6120015f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2092086614

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614) 
   * a668de4b47df64e2d09b8c1bd0a172271c41a7e3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2095279710

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662",
       "triggerID" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4c6c60d3b1bdde2c780374b44962e77a6120015f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23683",
       "triggerID" : "4c6c60d3b1bdde2c780374b44962e77a6120015f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * adc1380cb496881fd2f1c8b30aa059759c7c5c9c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662) 
   * 4c6c60d3b1bdde2c780374b44962e77a6120015f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23683) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #11137:
URL: https://github.com/apache/hudi/pull/11137


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590551059


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -167,20 +167,19 @@ public void testMetadataBootstrapInflightCommit() throws Exception {
     HoodieTableType tableType = COPY_ON_WRITE;
     init(tableType, false);
 
+    // In real production env, bootstrap action can only happen on empty table,
+    // otherwise we need to roll back the previous bootstrap first,
+    // see 'SparkBootstrapCommitActionExecutor.execute' for more details.
     doPreBootstrapWriteOperation(testTable, INSERT, "0000001");
     doPreBootstrapWriteOperation(testTable, "0000002");
     // add an inflight commit
     HoodieCommitMetadata inflightCommitMeta = testTable.doWriteOperation("00000007", UPSERT, emptyList(),
-        asList("p1", "p2"), 2, true, true);
+        asList("p1", "p2"), 2, false, true);

Review Comment:
   I added the comment, the bootstrap action only happens once, that is the limination of itself, the tricky part is that the commit metadata for bootstrap file metadata only contains the file group id instead of specfic file names.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590555917


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -261,12 +260,8 @@ private void bootstrapAndVerifyFailure() throws Exception {
     writeConfig = getWriteConfig(true, true);
     initWriteConfigAndMetatableWriter(writeConfig, true);
     syncTableMetadata(writeConfig);
-    try {
-      validateMetadata(testTable);
-      Assertions.fail("Should have failed");
-    } catch (IllegalStateException e) {
-      // expected
-    }
+    Assertions.assertThrows(AssertionFailedError.class, () -> validateMetadata(testTable),

Review Comment:
   I just revert it back to what it was.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2090242787

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2090230760

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11137:
URL: https://github.com/apache/hudi/pull/11137#issuecomment-2094535016

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614",
       "triggerID" : "fe7584c435e0ba03e9176cf5e7cc331d9a0052d7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644",
       "triggerID" : "a668de4b47df64e2d09b8c1bd0a172271c41a7e3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662",
       "triggerID" : "adc1380cb496881fd2f1c8b30aa059759c7c5c9c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a668de4b47df64e2d09b8c1bd0a172271c41a7e3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644) 
   * adc1380cb496881fd2f1c8b30aa059759c7c5c9c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23662) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590550442


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -925,6 +925,37 @@ public void testMetadataTableCompactionWithPendingInstants() throws Exception {
     assertEquals(HoodieInstantTimeGenerator.instantTimeMinusMillis(inflightInstant2, 1L), tableMetadata.getLatestCompactionTime().get());
   }
 
+  @Test
+  public void testInitializeMetadataTableWithPendingInstant() throws Exception {
+    init(COPY_ON_WRITE, false);

Review Comment:
   There should be no much difference for the table type, maybe we can just paramatize this test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11137:
URL: https://github.com/apache/hudi/pull/11137#discussion_r1590555917


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieMetadataBootstrap.java:
##########
@@ -261,12 +260,8 @@ private void bootstrapAndVerifyFailure() throws Exception {
     writeConfig = getWriteConfig(true, true);
     initWriteConfigAndMetatableWriter(writeConfig, true);
     syncTableMetadata(writeConfig);
-    try {
-      validateMetadata(testTable);
-      Assertions.fail("Should have failed");
-    } catch (IllegalStateException e) {
-      // expected
-    }
+    Assertions.assertThrows(AssertionFailedError.class, () -> validateMetadata(testTable),

Review Comment:
   I just revert it back to what it was.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org