You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "majian1998 (via GitHub)" <gi...@apache.org> on 2023/08/18 04:45:36 UTC

[GitHub] [hudi] majian1998 opened a new pull request, #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

majian1998 opened a new pull request, #9472:
URL: https://github.com/apache/hudi/pull/9472

   ### Change Logs
   Implemented a solution to prevent duplicate key errors in concurrent replace commit operations.
   Registered the replace file ID information to the timeline for each replace commit, whether it is submitted or not.
   Saved the pending replace commit's partitionToReplaceFileIds information to the inflight commit's extra metadata.
   Updated drop partition and clustering operations to read the partitionToReplaceFileIds information in the timeline to ensure no duplicate file IDs.
   Removed the modification involved in the related issue for compaction commit.
   
   ### Impact
   
   No public API or user-facing feature changes.
   
   ### Risk level (write none, low medium or high below)
   
   low
   
   ### Documentation Update
   
   Related issue: https://issues.apache.org/jira/browse/HUDI-5553
   
   The specific problem is that when concurrent replace commit operations are executed, two replace commits may point to the same file ID, resulting in a duplicate key error. The existing issue solved the problem of scheduling delete partition while there are pending clustering or compaction operations, which will be prevented in this case. However, this solution is not perfect and may still cause data inconsistency if a clustering plan is scheduled before the delete partition is committed. Because validation is one-way.In this case, both replace commits will still contain duplicate keys, and the table will become inconsistent when both plans are committed. This is very fatal, and there are other similar scenarios that may bypass the validation of the existing issue. Moreover, the existing issue is at the partition level and is not precise enough.
   
   Here is my solution:
   ![image](https://github.com/apache/hudi/assets/47964462/6d8a3134-96a5-45ec-8ed0-ed2776b7ed24)
   
   As shown in the figure, both drop partition and clustering will go through a period of time that is not registered to the timeline, which is the scenario that the previous issue did not solve. Here, I register the replace file IDs involved in each replace commit to the active timeline (the replace commit timeline that has been submitted has saved partitionToReplaceFileIds, and only pending requests need to be processed). Since in the case of Spark SQL, delete partition creates a requested commit in advance during write, which is inconvenient to handle, I save the pending replace commit's partitionToReplaceFileIds information to the inflight commit's extra metadata. Therefore, each time drop partition or clustering is executed, it only needs to read the partitionToReplaceFileIds information in the timeline after ensuring that the inflight commit information has been saved to the timeline to ensure that there are no duplicate file IDs and prevent this kind of error from occurring.
   
   In simple terms, each replace commit will register the replace file ID information to the timeline whether it is submitted or not, at the same time, each submission will check this information to ensure that it will not be repeated, so that any replace commit containing this file ID will be prevented, ensuring that there are no duplicate keys.
   
   When this idea is also implemented on the compaction commit, the modification involved in the related issue can be removed.
   
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683379797

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dba536eaf3fbc3cade137d7c9d24c705e8263ad9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683464086

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dba536eaf3fbc3cade137d7c9d24c705e8263ad9 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346) 
   * 13a06dff7a03d861232980b79baf924e31d55ff7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1687555980

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398) 
   * bf77139cceeae0254884dfb53a1379aadb8c7316 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685802938

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298274999


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {

Review Comment:
   Yes, in the pending timeline  I only saved the corresponding information in infight replace commits. This information is not saved in the requested commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683852664

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38915491149644dd98eafc809e7cecb673524eb4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352) 
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1303744014


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";

Review Comment:
   I don't think this situation will happen:
   Because my validation was done after the inflight commit was written to disk. Suppose two inflight replace commits happen at the same time:
   1. Simultaneous disk write: Even if the disk write happens at the same time, when validation begins, it will reload first, so it will definitely see the file that the other party has written to disk.
   2. Simultaneous validation: This is even if both have written to disk, both validation will find each other and report errors.
   3. One party validates while the other writes to disk: This means that the verifying party has already completed the disk write, so the committing party will definitely be found.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305191846


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file ID, which will cause the table status to be abnormal. The replace commit has been terminated.");

Review Comment:
   This is a great suggestion, but would printing all file IDs in case of multiple conflicts make the log bloated and difficult to read? Alternatively, could I perform some additional processing in this stream to pass along the instant and print the conflicting instant instead? However, adding more processing may slightly reduce the readability of this stream.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1687939317

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * bf77139cceeae0254884dfb53a1379aadb8c7316 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683706780

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 13a06dff7a03d861232980b79baf924e31d55ff7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348) 
   * 38915491149644dd98eafc809e7cecb673524eb4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1684167747

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685622743

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1684902865

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300999330


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {

Review Comment:
   This is because in the current implementation, the inflight and completed statuses are judged based on reading different content.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition. [hudi]

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 closed pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.
URL: https://github.com/apache/hudi/pull/9472


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1684878994

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1686459322

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1688218200

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf77139cceeae0254884dfb53a1379aadb8c7316 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402) 
   * 7e2d64b9b04ee6248c75c749ed4583954dd3c583 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685225421

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377) 
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1698545630

   TODO: Implement additional validations for insert overwrite, compaction, and these operations on Flink.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1693079996

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417",
       "triggerID" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1264a3a18c712b6ab80e7256d628c46338062433",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19460",
       "triggerID" : "1264a3a18c712b6ab80e7256d628c46338062433",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1264a3a18c712b6ab80e7256d628c46338062433 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19460) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305077926


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);

Review Comment:
   NIT: ~~replace commit~~ ->  `REPLACECOMMIT`



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file ID, which will cause the table status to be abnormal. The replace commit has been terminated.");

Review Comment:
   ~~Replace commit~~ -> `REPLACECOMMIT`
   ~~replace commit~~ -> `REPLACECOMMIT`
   
   Log the offending file ID too? Since this validation will mainly affect jobs with high concurrency/multi-writer, it's the offending file ID will allow users to easily find which (other) job it is conflicting with and "tune" it accordingly if required. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1692857813

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417",
       "triggerID" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1264a3a18c712b6ab80e7256d628c46338062433",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1264a3a18c712b6ab80e7256d628c46338062433",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d403d1696e76ee6daf6ad6ddc62e9a218a5accab Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417) 
   * 1264a3a18c712b6ab80e7256d628c46338062433 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305217059


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file ID, which will cause the table status to be abnormal. The replace commit has been terminated.");

Review Comment:
   done~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305231230


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading REPLACECOMMIT meta", e);
+            return null;

Review Comment:
   In what cases are we not able to read the REPLACECOMMIT meta here? If we are not able to read the meta here, is it safe to proceed with the REPLACECOMMIT that the current writer is performing?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1692866533

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417",
       "triggerID" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1264a3a18c712b6ab80e7256d628c46338062433",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19460",
       "triggerID" : "1264a3a18c712b6ab80e7256d628c46338062433",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d403d1696e76ee6daf6ad6ddc62e9a218a5accab Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417) 
   * 1264a3a18c712b6ab80e7256d628c46338062433 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19460) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] stream2000 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300966820


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {

Review Comment:
   Can we remove this check since we have already filtered out non-inflight instants? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683909023

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 38915491149644dd98eafc809e7cecb673524eb4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352) 
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1686215731

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1686257671

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388) 
   * 1e493605d0a26b442efbf1518b063dbb1e616872 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1686382580

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition. [hudi]

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1782170663

   @bvaradar Your suggestion is reasonable, as it allows detecting most conflict situations at a lower cost. 
   However, there may still be some issues with this PR. If we check for conflicts before generating the requested files, it won't completely avoid situations where two processes simultaneously reach this point without being aware of each other. On the other hand, if we check for conflicts after generating the requested files, and let the timeline act as a marker, even if we detect conflicts and terminate the final execution process, there will still be failed requested files left to handle, which might not be ideal.
   What is your perspective on this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] boneanxs commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298256125


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {

Review Comment:
   Only need inflight replace commits?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {

Review Comment:
   Understand the intension that deserialize commit might be costly, but I prefer sequential stream instead of parallel. the parallelism is out of control and usually the merging is costly.
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java:
##########
@@ -238,10 +248,23 @@ protected abstract Iterator<List<WriteStatus>> handleUpdate(String partitionPath
 
   protected HoodieWriteMetadata<HoodieData<WriteStatus>> executeClustering(HoodieClusteringPlan clusteringPlan) {

Review Comment:
   Looks this is only fixed inside clustering/delete, is it possible 2 concurrent `insert overwrite` queries also has such issue?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {
+        HoodieReplaceCommitMetadata replaceCommitMetadata =
+            HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+        if (!instant.isCompleted()) {
+          return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault("replaceCommitFileIds", ""), String[].class);
+        } else {
+          return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+              .flatMap(List::stream)
+              .toArray(String[]::new);
+        }
+      } catch (IOException e) {

Review Comment:
   We should throw the error out also?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {
+        HoodieReplaceCommitMetadata replaceCommitMetadata =
+            HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+        if (!instant.isCompleted()) {
+          return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault("replaceCommitFileIds", ""), String[].class);
+        } else {

Review Comment:
   Better make this "replaceCommitFileIds" a static constant



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298270439


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java:
##########
@@ -238,10 +248,23 @@ protected abstract Iterator<List<WriteStatus>> handleUpdate(String partitionPath
 
   protected HoodieWriteMetadata<HoodieData<WriteStatus>> executeClustering(HoodieClusteringPlan clusteringPlan) {

Review Comment:
   Yes, there will be such problems in insert overwrite and compaction. At present, it is only implemented in clustering. After ensuring the correct idea, it will be extended to other actions.Then we can ignore the validation by HUDI-5553



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685226953

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377) 
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300982795


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (IOException e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file id!");

Review Comment:
   Reply “I think we better throw the exception here instead of return null”:
   I made the decision based on the following consideration: there may be some ways, such as constructing tests that do not create an inflight status timeline through normal execution processes. My additional validation information will not be added here, but if such a situation occurs during execution, an error will be reported here due to parsing exceptions. I do not want the additional validation procedures introduced here to affect the original logic. Without my validation information, the replace function will not be executed properly. Therefore, I only report the error through logging, but do not throw an exception. Do you think this makes sense?
   
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (IOException e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file id!");

Review Comment:
   Reply “I think we better throw the exception here instead of return null”:
   I made the decision based on the following consideration: there may be some ways, such as constructing tests that do not create an inflight status timeline through normal execution processes. My additional validation information will not be added here, but if such a situation occurs during execution, an error will be reported here due to parsing exceptions. I do not want the additional validation procedures introduced here to affect the original logic. Without my validation information, the replace function will not be executed properly. Therefore, I only report the error through logging, but do not throw an exception. Do you think this makes sense?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300994921


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";

Review Comment:
   This is because it appears that there is no relevant file ID information in the inflight status. However, if there are two plans being executed simultaneously and not yet submitted, can we prevent this situation from occurring by performing additional validation on the inflight status?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1689322594

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417",
       "triggerID" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7e2d64b9b04ee6248c75c749ed4583954dd3c583 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409) 
   * d403d1696e76ee6daf6ad6ddc62e9a218a5accab Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1689316443

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7e2d64b9b04ee6248c75c749ed4583954dd3c583 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409) 
   * d403d1696e76ee6daf6ad6ddc62e9a218a5accab UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305249985


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading REPLACECOMMIT meta", e);
+            return null;

Review Comment:
   The issue currently observed is that the key "REPLACE_COMMIT_FILE_IDS" does not have a value in the inflight metadata. In such cases, the code throws an error, but in certain testing scenarios, it may not call the generated inflight REPLACECOMMIT through `table.cluster.` Instead, it directly calls transitionReplaceRequestedToInflight to generate the inflight operation. Consequently, an exception occurs because REPLACE_COMMIT_FILE_IDS is not stored in the inflight metadata by `table.cluster`. No other issues have been identified thus far. The reason for catching this exception without terminating the program is that it serves as an additional validation program. I do not want to impact the original execution logic unless there is a validation failure. Therefore, any other unexpected exceptions are also skipped in this scenario. We can discuss how this situation can be handled better.  ^_^



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition. [hudi]

Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1783579700

   @majian1998 : The conflict resolution happens within the context of LockManager transaction which should serialize the checking. From what I understand, the conflict resolution strategy needs to handle replace commits. We need to incorporate the checking similar to DeletePartitionUtils.checkForPendingTableServiceActions as part of conflict resolution. Currently, DeletePartitionUtils.checkForPendingTableServiceActions is being called outside of transaction scope causing this. Let me know if this makes sense. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] boneanxs commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298294422


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {

Review Comment:
   Parallel stream requires a merge operation, and usually it could be costly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685262390

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300983273


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (IOException e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file id!");

Review Comment:
   I will revise this description



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1687573104

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398) 
   * bf77139cceeae0254884dfb53a1379aadb8c7316 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] stream2000 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300967476


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);

Review Comment:
   use get instead and deal throw a readable exception when the `REPLACE_COMMIT_FILE_IDS` is null



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305179990


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {

Review Comment:
   done 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685165782

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1304195108


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);

Review Comment:
   Thanks for your suggestion! I think your suggestion is very helpful, and if the details are implemented, it can indeed solve the same problem.
   However, my implementation is actually slightly different. I believe my implementation will be more lightweight, as it won't rely on the OCC framework and won't require locking. If locks are involved in the replace commit, do we still need to go through a series of lock configurations? 
   Moreover, I think OCC solves the problem of rolling back when two files modify the same file id, which can be optional. On the other hand, my solution addresses the problem of two replace commits simultaneously deleting the same file id. If this situation occurs, it will result in incorrect table state, and the entire table will be unable to perform any operations, such as queries resulting in errors. Therefore, I don't think this check should be categorized as an optional OCC write check, but should be enabled at all times.
   Regarding the issue mentioned by @boneanxs , I have already addressed it, and I don't think it will occur under my check.
   Do you think my modification can achieve the effect I mentioned?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] boneanxs commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1303729529


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";

Review Comment:
   How you validate it if such issue happens? Like 2 inflight commits written nearly at the same time, and each other doesn't see it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298276321


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {
+        HoodieReplaceCommitMetadata replaceCommitMetadata =
+            HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+        if (!instant.isCompleted()) {
+          return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault("replaceCommitFileIds", ""), String[].class);
+        } else {

Review Comment:
   fine!



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {
+        HoodieReplaceCommitMetadata replaceCommitMetadata =
+            HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+        if (!instant.isCompleted()) {
+          return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault("replaceCommitFileIds", ""), String[].class);
+        } else {
+          return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+              .flatMap(List::stream)
+              .toArray(String[]::new);
+        }
+      } catch (IOException e) {

Review Comment:
   It was my oversight, I have modified this part



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683841975

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 13a06dff7a03d861232980b79baf924e31d55ff7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348) 
   * 38915491149644dd98eafc809e7cecb673524eb4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685223928

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685615611

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1686244791

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] stream2000 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300967476


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);

Review Comment:
   use get instead and throw a readable exception when the `REPLACE_COMMIT_FILE_IDS` is null



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] boneanxs commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300858914


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (IOException e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file id!");

Review Comment:
   Can make the exception more clear



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (IOException e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file id!");
+              });
+        });
+  }
+}

Review Comment:
   Missing empty line here



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {
+        HoodieReplaceCommitMetadata replaceCommitMetadata =
+            HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+        if (!instant.isCompleted()) {
+          return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault("replaceCommitFileIds", ""), String[].class);
+        } else {
+          return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+              .flatMap(List::stream)
+              .toArray(String[]::new);
+        }
+      } catch (IOException e) {

Review Comment:
   I think we better throw the exception here instead of return null



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {

Review Comment:
   `getInstants().stream()` -> `getInstantsAsStream()`



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";

Review Comment:
   I'm thinking we can directly use `replaceCommitMetadata.partitionToReplaceFileIds` given the inflight replace commits actually don't use it. This way we don't need to validate it differently for `inflight` and `complete`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition. [hudi]

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1784704711

   @bvaradar: I believe that DeletePartitionUtils.checkForPendingTableServiceActions prevents potential conflicts from occurring earlier. Of course, if this cannot guarantee perfect conflict resolution, I think it is acceptable to sacrifice these performance gains. Detecting conflicts within the transaction does ensure that conflict detection is executed in a serialized manner.
   
   However, I still have two questions that I would like to ask.
   1.Is performing partition-level conflict detection before ending the transaction a more concise and reasonable modification for this PR?
   2.When a conflict occurs, should we directly rollback the conflicting instant and display a message, or should we simply throw an error? Are there any instants where rolling back is not suitable? For example, in the case of consistency hashing clustering.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300982366


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {

Review Comment:
   👌



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {

Review Comment:
   👌



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1686443773

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388) 
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1687403746

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1687860429

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305067820


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {

Review Comment:
   NIT: Use a `flatMap` here instead? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] voonhous commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305209537


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file ID, which will cause the table status to be abnormal. The replace commit has been terminated.");

Review Comment:
   The first file id should be enough. (The `s` variable here should be enough)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1688233766

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf77139cceeae0254884dfb53a1379aadb8c7316 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402) 
   * 7e2d64b9b04ee6248c75c749ed4583954dd3c583 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1698548666

   @boneanxs @TengHuo @voonhous  have any other comments or questions for this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683619955

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 13a06dff7a03d861232980b79baf924e31d55ff7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1684711609

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685163877

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1686331544

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388) 
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1684944263

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685220056

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1300983456


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (IOException e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file id!");
+              });
+        });
+  }
+}

Review Comment:
   👌



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (IOException e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading replace commit meta", e);
+            return null;
+          }
+        }).filter(Objects::nonNull)
+        .forEach(fileIdArray -> {
+          Arrays.stream(fileIdArray)
+              .filter(fileId -> !replaceFileids.add(fileId))
+              .findFirst()
+              .ifPresent(s -> {
+                throw new HoodieException("Replace commit involves duplicate file id!");
+              });
+        });
+  }
+}

Review Comment:
   👌



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1687926779

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398) 
   * bf77139cceeae0254884dfb53a1379aadb8c7316 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683373888

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dba536eaf3fbc3cade137d7c9d24c705e8263ad9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] boneanxs commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298293054


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {

Review Comment:
   But currently you includes all inflight commits, not just inflight replace commits.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683803707

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298273081


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {

Review Comment:
   I think you are right. I will remove the parallelism. And "the merging is costly" refers to the concat operation? I do not quite understand.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1688574664

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 7e2d64b9b04ee6248c75c749ed4583954dd3c583 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1684902220

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1687273434

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1685199608

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1301002942


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);

Review Comment:
   👌



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstants().stream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().getOrDefault(REPLACE_COMMIT_FILE_IDS, "non-existent key"), String[].class);

Review Comment:
   👌



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] TengHuo commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "TengHuo (via GitHub)" <gi...@apache.org>.
TengHuo commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1303746084


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);

Review Comment:
   I think this issue is caused by different writers isolation from each other, and there is no lock in Hudi timeline, so when one writer is trying to `validateReplaceCommit`, it's impossible to guarantee that there is no other writer is modifying the same timeline.
   
   As @boneanxs mentioned, there could be other situation such as two `insert overwrite` running together, or `compaction` and `clustering` generating plan at the same time, the current timeline can't provide enough information (any partitions modified or dropped) to each writer to prevent these concurrent issue.
   
   In Hudi [RFC-56](https://github.com/apache/hudi/blob/master/rfc/rfc-56/rfc-56.md), OCC was introduced for solving write conflicts in file group level, but no conflict check in partition level. So I think this partition level conflict issue should be solved in OCC framework, it's better because all similar situation can be solved as long as OCC enabled.
   
   cc @voonhous 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1686822578

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1304205434


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);

Review Comment:
   In short, I think OCC is an option that can be enabled or disabled, and disabling it won't result in serious errors. However, in the scenario of my check, if it is not detected, it will lead to table errors. Therefore, it should not be controlled by the same switch as OCC. Additionally, my check is also low-overhead and does not introduce any other dependencies.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1689626305

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19409",
       "triggerID" : "7e2d64b9b04ee6248c75c749ed4583954dd3c583",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417",
       "triggerID" : "d403d1696e76ee6daf6ad6ddc62e9a218a5accab",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d403d1696e76ee6daf6ad6ddc62e9a218a5accab Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19417) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1687279411

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 1e493605d0a26b442efbf1518b063dbb1e616872 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1688098810

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19365",
       "triggerID" : "1684902220",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19376",
       "triggerID" : "1685163877",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19377",
       "triggerID" : "1685220056",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19379",
       "triggerID" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19383",
       "triggerID" : "1685615611",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c0019c0fc1d1803b9e0ccfbd1c9de953d6aba4f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19388",
       "triggerID" : "1686215731",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19390",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19389",
       "triggerID" : "1686382580",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687273434",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e493605d0a26b442efbf1518b063dbb1e616872",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19398",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "bf77139cceeae0254884dfb53a1379aadb8c7316",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402",
       "triggerID" : "1687860429",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * bf77139cceeae0254884dfb53a1379aadb8c7316 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19405) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19402) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1305253669


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static final String REPLACE_COMMIT_FILE_IDS = "replaceCommitFileIds";
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstantsAsStream(),
+        metaClient.getActiveTimeline().filterInflights().filterPendingReplaceTimeline().getInstantsAsStream()).map(instant -> {
+          try {
+            HoodieReplaceCommitMetadata replaceCommitMetadata =
+                HoodieReplaceCommitMetadata.fromBytes(metaClient.getActiveTimeline().getInstantDetails(instant).get(),
+                HoodieReplaceCommitMetadata.class);
+            if (!instant.isCompleted()) {
+              return JsonUtils.getObjectMapper().readValue(replaceCommitMetadata.getExtraMetadata().get(REPLACE_COMMIT_FILE_IDS), String[].class);
+            } else {
+              return replaceCommitMetadata.getPartitionToReplaceFileIds().values().stream()
+                  .flatMap(List::stream)
+                  .toArray(String[]::new);
+            }
+          } catch (Exception e) {
+            // If the key does not exist or there is a JSON parsing error, LOG reports an error and ignores it.
+            LOG.error("Error when reading REPLACECOMMIT meta", e);
+            return null;

Review Comment:
   In other words, my expectation is that the program should not encounter any issues leading to termination in my validation, unless there are duplicate file ids.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1684732422

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19354",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "1683803707",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355",
       "triggerID" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f0de8a6d00fe72108a12d8316cb1d38389d6b31",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362",
       "triggerID" : "1684711609",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 4f0de8a6d00fe72108a12d8316cb1d38389d6b31 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19355) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19362) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683419217

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * dba536eaf3fbc3cade137d7c9d24c705e8263ad9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346) 
   * 13a06dff7a03d861232980b79baf924e31d55ff7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298301510


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {

Review Comment:
   Thanks for your opinion, I understand now!



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {
+      try {

Review Comment:
   Thanks for your opinion, I understand now!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9472:
URL: https://github.com/apache/hudi/pull/9472#issuecomment-1683715844

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19346",
       "triggerID" : "dba536eaf3fbc3cade137d7c9d24c705e8263ad9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348",
       "triggerID" : "13a06dff7a03d861232980b79baf924e31d55ff7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38915491149644dd98eafc809e7cecb673524eb4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352",
       "triggerID" : "38915491149644dd98eafc809e7cecb673524eb4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 13a06dff7a03d861232980b79baf924e31d55ff7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19348) 
   * 38915491149644dd98eafc809e7cecb673524eb4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19352) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] majian1998 commented on a diff in pull request #9472: [HUDI-6719]Fix data inconsistency issues caused by concurrent clustering and delete partition.

Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on code in PR #9472:
URL: https://github.com/apache/hudi/pull/9472#discussion_r1298302932


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {

Review Comment:
   My fault🤦‍♂️   I overlooked this, I will fix it



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ReplaceCommitValidateUtil.java:
##########
@@ -0,0 +1,49 @@
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.util.JsonUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Stream;
+
+import static org.apache.hudi.common.util.FileIOUtils.LOG;
+
+public class ReplaceCommitValidateUtil {
+  public static void validateReplaceCommit(HoodieTableMetaClient metaClient) {
+    metaClient.reloadActiveTimeline();
+    Set<String> replaceFileids = new HashSet<>();
+
+    // Verify pending and completed replace commit
+    Stream.concat(metaClient.getActiveTimeline().getCompletedReplaceTimeline().getInstants().stream(),
+        metaClient.getActiveTimeline().filterInflights().getInstants().stream()).parallel().map(instant -> {

Review Comment:
   My fault🤦‍♂️   I overlooked this, I will fix it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org