You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "stream2000 (via GitHub)" <gi...@apache.org> on 2023/05/18 02:32:34 UTC

[GitHub] [hudi] stream2000 opened a new pull request, #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

stream2000 opened a new pull request, #8745:
URL: https://github.com/apache/hudi/pull/8745

   …ion in multi writer scenario
   
   
   see relative issue #8500
   
   ### Change Logs
   
   _Describe context and summary for this change. Highlight if any code was copied._
   
   ### Impact
   
   _Describe any public API or user-facing feature change or any performance impact._
   
   ### Risk level (write none, low medium or high below)
   
   _If medium or high, explain what verification was done to mitigate the risks._
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change_
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1218862785


##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -298,6 +298,22 @@ protected void syncHoodieTable(String tableName, boolean useRealtimeInputFormat,
     LOG.info("Sync complete for " + tableName);
   }
 
+  private boolean needToSyncAllPartitions(Option<String> lastCommitTimeSynced) {
+    if (!lastCommitTimeSynced.isPresent()) {
+      return true;
+    }
+    if (config.getBoolean(META_SYNC_USE_STATE_TRANSIENT_TIME)) {
+      // If we use state transient time to sync partitions and the last commit time synced is before latest archive time
+      // We need to fall back to list all partitions instead of load the whole archive timeline
+      Option<String> latestArchiveTime = syncClient.getLastArchiveTime();

Review Comment:
   > the archiver will archive it immediately. And if we just looking the active timelime to find the 'hollow' instants we will miss the instant just have been archived
   
   There is no good manner to resolve the problem at this stage for Hudi, because we have no effeicent scanning and filtering on the archived timeline, what we can address in the PR is the multi-writer scenarios, e.g., an instant which is inflight first whith following complete instants and then finishes, without the fix, it could be skipped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1553086225

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5efbdd6b1101dd41da0a12b32000430dc9d8d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1564018976

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5efbdd6b1101dd41da0a12b32000430dc9d8d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174) 
   * 4b220002d4f18a79806c8177a503b8e8ec66f9b2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1578370482

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * 53259c48a7165424ddf97c99a624aeb6644338c8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631) 
   * 7cac64292aa0cd454e6ef4162f754b8badffafe0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] stream2000 commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1581766358

   @danny0405 All comments addressed. PTAL, thank~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 merged pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8745:
URL: https://github.com/apache/hudi/pull/8745


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] stream2000 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1219271382


##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -298,6 +298,22 @@ protected void syncHoodieTable(String tableName, boolean useRealtimeInputFormat,
     LOG.info("Sync complete for " + tableName);
   }
 
+  private boolean needToSyncAllPartitions(Option<String> lastCommitTimeSynced) {
+    if (!lastCommitTimeSynced.isPresent()) {
+      return true;
+    }
+    if (config.getBoolean(META_SYNC_USE_STATE_TRANSIENT_TIME)) {
+      // If we use state transient time to sync partitions and the last commit time synced is before latest archive time
+      // We need to fall back to list all partitions instead of load the whole archive timeline
+      Option<String> latestArchiveTime = syncClient.getLastArchiveTime();

Review Comment:
   > what we can address in the PR is the multi-writer scenarios, e.g., an instant which is inflight first whith following complete instants and then finishes, without the fix, it could be skipped.
   
   Got it. Have updated the pr that no longer deals with the archive timeline. PTAL~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1218862785


##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -298,6 +298,22 @@ protected void syncHoodieTable(String tableName, boolean useRealtimeInputFormat,
     LOG.info("Sync complete for " + tableName);
   }
 
+  private boolean needToSyncAllPartitions(Option<String> lastCommitTimeSynced) {
+    if (!lastCommitTimeSynced.isPresent()) {
+      return true;
+    }
+    if (config.getBoolean(META_SYNC_USE_STATE_TRANSIENT_TIME)) {
+      // If we use state transient time to sync partitions and the last commit time synced is before latest archive time
+      // We need to fall back to list all partitions instead of load the whole archive timeline
+      Option<String> latestArchiveTime = syncClient.getLastArchiveTime();

Review Comment:
   > the archiver will archive it immediately. And if we just looking the active timelime to find the 'hollow' instants we will miss the instant just have been archived
   
   There is no good manner to resolve the problem at this stage for Hudi, because we have no efficient scanning and filtering on the archived timeline, what we can address in the PR is the multi-writer scenarios, e.g., an instant which is inflight first whith following complete instants and then finishes, without the fix, it could be skipped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] stream2000 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1197325515


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -234,6 +234,21 @@ public static HoodieTimeline getCommitsTimelineAfter(
     return timeline.getCommitsTimeline()
         .findInstantsAfter(exclusiveStartInstantTime, Integer.MAX_VALUE);
   }
+
+  /**
+   * Returns a Hudi timeline with commits after the given instant state transient time (exclusive).
+   *
+   * @param metaClient                {@link HoodieTableMetaClient} instance.
+   * @param exclusiveStartStateTransientTime Start instant time (exclusive).
+   * @return Hudi timeline.
+   */
+  public static HoodieTimeline getCommitsTimelineAfterStateTransientTime(
+      HoodieTableMetaClient metaClient, String exclusiveStartStateTransientTime) {
+    HoodieActiveTimeline timeline = metaClient.getActiveTimeline();
+    // TODO: Need to add support for archive timeline

Review Comment:
   We need to get instants with greater state transient time in archive time in an efficient way, otherwise we need to load the whole archive timeline here. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1564107622

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4b220002d4f18a79806c8177a503b8e8ec66f9b2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399) 
   * 5b623fd433f9e52d0464f82fda115a08353da6da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1552337400

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5efbdd6b1101dd41da0a12b32000430dc9d8d460 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1564085293

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5efbdd6b1101dd41da0a12b32000430dc9d8d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174) 
   * 4b220002d4f18a79806c8177a503b8e8ec66f9b2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1578151906

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62377696531fc1d4ee2b7c0c86897d1cfb6b5de9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556) 
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1571723780

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5b623fd433f9e52d0464f82fda115a08353da6da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402) 
   * 62377696531fc1d4ee2b7c0c86897d1cfb6b5de9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1578357829

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62377696531fc1d4ee2b7c0c86897d1cfb6b5de9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556) 
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * 53259c48a7165424ddf97c99a624aeb6644338c8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631) 
   * 7cac64292aa0cd454e6ef4162f754b8badffafe0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1578639760

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635",
       "triggerID" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * 7cac64292aa0cd454e6ef4162f754b8badffafe0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633) 
   * e4a9743db12add71510fd6f40e70d73dc12213b3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1220821891


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -223,17 +225,24 @@ public static HoodieDefaultTimeline getTimeline(HoodieTableMetaClient metaClient
    * @param exclusiveStartInstantTime Start instant time (exclusive).
    * @return Hudi timeline.
    */
-  public static HoodieTimeline getCommitsTimelineAfter(
+  public static HoodieDefaultTimeline getCommitsTimelineAfter(
       HoodieTableMetaClient metaClient, String exclusiveStartInstantTime) {

Review Comment:
   Return `HoodieTimeline` instead.



##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -341,8 +353,15 @@ public void close() {
   @Override
   public void updateLastCommitTimeSynced(String tableName) {
     // Set the last commit time from the TBLproperties
-    Option<String> lastCommitSynced = getActiveTimeline().lastInstant().map(HoodieInstant::getTimestamp);
-    if (lastCommitSynced.isPresent()) {
+    HoodieTimeline activeTimeline = getActiveTimeline();
+    Option<String> lastCommitSynced = activeTimeline.lastInstant().map(HoodieInstant::getTimestamp);
+    Option<String> lastCommitCompletionSynced = getActiveTimeline()
+        .getInstantsOrderedByStateTransitionTime()
+        .skip(activeTimeline.countInstants() - 1)
+        .findFirst()
+        .map(i -> Option.of(i.getStateTransitionTime()))
+        .orElse(Option.empty());
+    if (lastCommitSynced.isPresent() && lastCommitCompletionSynced.isPresent()) {
       try {

Review Comment:
   A decision for `lastCommitSynced.isPresent()` should be enough.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -223,17 +225,24 @@ public static HoodieDefaultTimeline getTimeline(HoodieTableMetaClient metaClient
    * @param exclusiveStartInstantTime Start instant time (exclusive).
    * @return Hudi timeline.
    */
-  public static HoodieTimeline getCommitsTimelineAfter(
+  public static HoodieDefaultTimeline getCommitsTimelineAfter(
       HoodieTableMetaClient metaClient, String exclusiveStartInstantTime) {
     HoodieActiveTimeline activeTimeline = metaClient.getActiveTimeline();
     HoodieDefaultTimeline timeline =
         activeTimeline.isBeforeTimelineStarts(exclusiveStartInstantTime)
             ? metaClient.getArchivedTimeline(exclusiveStartInstantTime)
             .mergeTimeline(activeTimeline)
             : activeTimeline;
-    return timeline.getCommitsTimeline()
+    return (HoodieDefaultTimeline) timeline.getCommitsTimeline()
         .findInstantsAfter(exclusiveStartInstantTime, Integer.MAX_VALUE);
   }
+
+  public static HoodieDefaultTimeline getHollowInstantsTimeline(
+      HoodieTableMetaClient metaClient, String exclusiveStartInstantTime, String commitCompletionTime) {

Review Comment:
   Return `HoodieTimeline` instead.



##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/HoodieSyncClient.java:
##########
@@ -126,16 +131,20 @@ public List<String> getAllPartitionPathsOnStorage() {
         config.getBoolean(META_SYNC_ASSUME_DATE_PARTITION));
   }
 
-  public List<String> getWrittenPartitionsSince(Option<String> lastCommitTimeSynced) {
+  public List<String> getWrittenPartitionsSince(Option<String> lastCommitTimeSynced, Option<String> lastCommitCompletionTimeSynced) {
     if (!lastCommitTimeSynced.isPresent()) {
       LOG.info("Last commit time synced is not known, listing all partitions in "
           + config.getString(META_SYNC_BASE_PATH)
           + ",FS :" + config.getHadoopFileSystem());
       return getAllPartitionPathsOnStorage();
     } else {
       LOG.info("Last commit time synced is " + lastCommitTimeSynced.get() + ", Getting commits since then");
-      return TimelineUtils.getWrittenPartitions(
-          TimelineUtils.getCommitsTimelineAfter(metaClient, lastCommitTimeSynced.get()));
+      HoodieDefaultTimeline writeTimeline = TimelineUtils.getCommitsTimelineAfter(metaClient, lastCommitTimeSynced.get());
+      if (lastCommitCompletionTimeSynced.isPresent()) {
+        writeTimeline = writeTimeline.mergeTimeline(TimelineUtils.getHollowInstantsTimeline(
+            metaClient, lastCommitTimeSynced.get(), lastCommitCompletionTimeSynced.get()));
+      }

Review Comment:
   Ditto.



##########
hudi-sync/hudi-sync-common/src/main/java/org/apache/hudi/sync/common/HoodieSyncClient.java:
##########
@@ -88,10 +89,14 @@ public boolean isBootstrap() {
    * If last sync time is not known then consider only active timeline.
    * Going through archive timeline is a costly operation, and it should be avoided unless some start time is given.
    */
-  public Set<String> getDroppedPartitionsSince(Option<String> lastCommitTimeSynced) {
-    HoodieTimeline timeline = lastCommitTimeSynced.isPresent()
+  public Set<String> getDroppedPartitionsSince(Option<String> lastCommitTimeSynced, Option<String> lastCommitCompletionTimeSynced) {
+    HoodieDefaultTimeline timeline = lastCommitTimeSynced.isPresent()
         ? TimelineUtils.getCommitsTimelineAfter(metaClient, lastCommitTimeSynced.get())
         : metaClient.getActiveTimeline();
+    timeline = lastCommitCompletionTimeSynced.isPresent()
+        ? timeline.mergeTimeline(TimelineUtils.getHollowInstantsTimeline(
+            metaClient, lastCommitTimeSynced.get(), lastCommitCompletionTimeSynced.get()))

Review Comment:
   Can we just fix the logic in `TimelineUtils.getCommitsTimelineAfter(` instead?



##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -341,8 +353,15 @@ public void close() {
   @Override
   public void updateLastCommitTimeSynced(String tableName) {
     // Set the last commit time from the TBLproperties
-    Option<String> lastCommitSynced = getActiveTimeline().lastInstant().map(HoodieInstant::getTimestamp);
-    if (lastCommitSynced.isPresent()) {
+    HoodieTimeline activeTimeline = getActiveTimeline();
+    Option<String> lastCommitSynced = activeTimeline.lastInstant().map(HoodieInstant::getTimestamp);
+    Option<String> lastCommitCompletionSynced = getActiveTimeline()
+        .getInstantsOrderedByStateTransitionTime()
+        .skip(activeTimeline.countInstants() - 1)

Review Comment:
   Does `.lastInstant()` work here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] stream2000 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1221421300


##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -341,8 +353,15 @@ public void close() {
   @Override
   public void updateLastCommitTimeSynced(String tableName) {
     // Set the last commit time from the TBLproperties
-    Option<String> lastCommitSynced = getActiveTimeline().lastInstant().map(HoodieInstant::getTimestamp);
-    if (lastCommitSynced.isPresent()) {
+    HoodieTimeline activeTimeline = getActiveTimeline();
+    Option<String> lastCommitSynced = activeTimeline.lastInstant().map(HoodieInstant::getTimestamp);
+    Option<String> lastCommitCompletionSynced = getActiveTimeline()
+        .getInstantsOrderedByStateTransitionTime()
+        .skip(activeTimeline.countInstants() - 1)

Review Comment:
   return value of `getInstantsOrderedByStateTransitionTime` is  `Stream<HoodieInstant>` and do not have method `lastInstant`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1580699408

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635",
       "triggerID" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0feaf9b421a770c43bf3035c08656101425ec049",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17651",
       "triggerID" : "0feaf9b421a770c43bf3035c08656101425ec049",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98ab9246c2e47a2b569ede46eed279000e34b11a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "98ab9246c2e47a2b569ede46eed279000e34b11a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * e4a9743db12add71510fd6f40e70d73dc12213b3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635) 
   * 0feaf9b421a770c43bf3035c08656101425ec049 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17651) 
   * 98ab9246c2e47a2b569ede46eed279000e34b11a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1578164464

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62377696531fc1d4ee2b7c0c86897d1cfb6b5de9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556) 
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * 53259c48a7165424ddf97c99a624aeb6644338c8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] stream2000 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1213840107


##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -298,6 +298,22 @@ protected void syncHoodieTable(String tableName, boolean useRealtimeInputFormat,
     LOG.info("Sync complete for " + tableName);
   }
 
+  private boolean needToSyncAllPartitions(Option<String> lastCommitTimeSynced) {
+    if (!lastCommitTimeSynced.isPresent()) {
+      return true;
+    }
+    if (config.getBoolean(META_SYNC_USE_STATE_TRANSIENT_TIME)) {
+      // If we use state transient time to sync partitions and the last commit time synced is before latest archive time
+      // We need to fall back to list all partitions instead of load the whole archive timeline
+      Option<String> latestArchiveTime = syncClient.getLastArchiveTime();

Review Comment:
   Maybe we need to scan some archive logs here, at least those created after `lastCommitTimeSynced`.  Otherwise, we need to sync all partitions every time we do the archive. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] stream2000 commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1572050050

   @danny0405 Hi danny, could you help to review this PR? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1552342516

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5efbdd6b1101dd41da0a12b32000430dc9d8d460 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1571736084

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5b623fd433f9e52d0464f82fda115a08353da6da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402) 
   * 62377696531fc1d4ee2b7c0c86897d1cfb6b5de9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1580718746

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635",
       "triggerID" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0feaf9b421a770c43bf3035c08656101425ec049",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17651",
       "triggerID" : "0feaf9b421a770c43bf3035c08656101425ec049",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98ab9246c2e47a2b569ede46eed279000e34b11a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17653",
       "triggerID" : "98ab9246c2e47a2b569ede46eed279000e34b11a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * 0feaf9b421a770c43bf3035c08656101425ec049 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17651) 
   * 98ab9246c2e47a2b569ede46eed279000e34b11a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17653) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1564096274

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5efbdd6b1101dd41da0a12b32000430dc9d8d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174) 
   * 4b220002d4f18a79806c8177a503b8e8ec66f9b2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399) 
   * 5b623fd433f9e52d0464f82fda115a08353da6da UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1578252439

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62377696531fc1d4ee2b7c0c86897d1cfb6b5de9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556) 
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * 53259c48a7165424ddf97c99a624aeb6644338c8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1579054284

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635",
       "triggerID" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * e4a9743db12add71510fd6f40e70d73dc12213b3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1553984735

   Ping me again when it is ready to reivew.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] stream2000 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1213873676


##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -298,6 +298,22 @@ protected void syncHoodieTable(String tableName, boolean useRealtimeInputFormat,
     LOG.info("Sync complete for " + tableName);
   }
 
+  private boolean needToSyncAllPartitions(Option<String> lastCommitTimeSynced) {
+    if (!lastCommitTimeSynced.isPresent()) {
+      return true;
+    }
+    if (config.getBoolean(META_SYNC_USE_STATE_TRANSIENT_TIME)) {
+      // If we use state transient time to sync partitions and the last commit time synced is before latest archive time
+      // We need to fall back to list all partitions instead of load the whole archive timeline
+      Option<String> latestArchiveTime = syncClient.getLastArchiveTime();

Review Comment:
   Thanks for your advice~Will take a look at that pr and see if we can get rid of introducing the config `META_SYNC_USE_STATE_TRANSIENT_TIME `



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1572190692

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 62377696531fc1d4ee2b7c0c86897d1cfb6b5de9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1564472398

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5b623fd433f9e52d0464f82fda115a08353da6da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1581034921

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635",
       "triggerID" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0feaf9b421a770c43bf3035c08656101425ec049",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17651",
       "triggerID" : "0feaf9b421a770c43bf3035c08656101425ec049",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98ab9246c2e47a2b569ede46eed279000e34b11a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17653",
       "triggerID" : "98ab9246c2e47a2b569ede46eed279000e34b11a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * 98ab9246c2e47a2b569ede46eed279000e34b11a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17653) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1580686730

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635",
       "triggerID" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0feaf9b421a770c43bf3035c08656101425ec049",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0feaf9b421a770c43bf3035c08656101425ec049",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * e4a9743db12add71510fd6f40e70d73dc12213b3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17635) 
   * 0feaf9b421a770c43bf3035c08656101425ec049 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8745:
URL: https://github.com/apache/hudi/pull/8745#issuecomment-1578527195

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17174",
       "triggerID" : "5efbdd6b1101dd41da0a12b32000430dc9d8d460",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17399",
       "triggerID" : "4b220002d4f18a79806c8177a503b8e8ec66f9b2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17402",
       "triggerID" : "5b623fd433f9e52d0464f82fda115a08353da6da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17556",
       "triggerID" : "62377696531fc1d4ee2b7c0c86897d1cfb6b5de9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e055d348ae5bef537653010b45e4ff9518e7aee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631",
       "triggerID" : "53259c48a7165424ddf97c99a624aeb6644338c8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633",
       "triggerID" : "7cac64292aa0cd454e6ef4162f754b8badffafe0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e4a9743db12add71510fd6f40e70d73dc12213b3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e055d348ae5bef537653010b45e4ff9518e7aee UNKNOWN
   * 53259c48a7165424ddf97c99a624aeb6644338c8 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17631) 
   * 7cac64292aa0cd454e6ef4162f754b8badffafe0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17633) 
   * e4a9743db12add71510fd6f40e70d73dc12213b3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] stream2000 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1218852843


##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -298,6 +298,22 @@ protected void syncHoodieTable(String tableName, boolean useRealtimeInputFormat,
     LOG.info("Sync complete for " + tableName);
   }
 
+  private boolean needToSyncAllPartitions(Option<String> lastCommitTimeSynced) {
+    if (!lastCommitTimeSynced.isPresent()) {
+      return true;
+    }
+    if (config.getBoolean(META_SYNC_USE_STATE_TRANSIENT_TIME)) {
+      // If we use state transient time to sync partitions and the last commit time synced is before latest archive time
+      // We need to fall back to list all partitions instead of load the whole archive timeline
+      Option<String> latestArchiveTime = syncClient.getLastArchiveTime();

Review Comment:
   @danny0405 Hi Danny, syncing both instant time and completion time to HMS will not help solve the archive issue. Assume we have an instant completed a long time after its instant time(for example 1 day), the archiver will archive it immediately. And if we just looking the active timelime to find the 'hollow' instants we will miss the instant just have been archived. To solve this problem we still need to know the max completion time in archive timeline anyway. What do you think? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on a diff in pull request #8745: [HUDI-6182] Hive sync use state transient time to avoid losing partit…

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8745:
URL: https://github.com/apache/hudi/pull/8745#discussion_r1213871041


##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:
##########
@@ -298,6 +298,22 @@ protected void syncHoodieTable(String tableName, boolean useRealtimeInputFormat,
     LOG.info("Sync complete for " + tableName);
   }
 
+  private boolean needToSyncAllPartitions(Option<String> lastCommitTimeSynced) {
+    if (!lastCommitTimeSynced.isPresent()) {
+      return true;
+    }
+    if (config.getBoolean(META_SYNC_USE_STATE_TRANSIENT_TIME)) {
+      // If we use state transient time to sync partitions and the last commit time synced is before latest archive time
+      // We need to fall back to list all partitions instead of load the whole archive timeline
+      Option<String> latestArchiveTime = syncClient.getLastArchiveTime();

Review Comment:
   I believe we can get rid of the config option `META_SYNC_USE_STATE_TRANSIENT_TIME` if we keep both the start time(instant time) and max completion time(transition time) in the HMS.
   
   We can use the instant time to check the max version id, and the completion time for real sync progress. Then we can fix the 'hollow' instants that are missed, just like what I fix in commit: https://github.com/apache/hudi/pull/8611



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org