You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "suryaprasanna (via GitHub)" <gi...@apache.org> on 2023/05/31 04:19:17 UTC

[GitHub] [hudi] suryaprasanna opened a new pull request, #8849: [UBER] Rollback enhancements

suryaprasanna opened a new pull request, #8849:
URL: https://github.com/apache/hudi/pull/8849

   ### Change Logs
   
   _Describe context and summary for this change. Highlight if any code was copied._
   
   ### Impact
   
   _Describe any public API or user-facing feature change or any performance impact._
   
   ### Risk level (write none, low medium or high below)
   
   _If medium or high, explain what verification was done to mitigate the risks._
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change_
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [HUDI-6304][UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1586059012

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741",
       "triggerID" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 912f9bdc6647cc7f3df090db001b9ec84dde0f53 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505) 
   * 2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [HUDI-6304] Handle cases when inflight files are removed and job failed leaving pending rollback files

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1586260595

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741",
       "triggerID" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17746",
       "triggerID" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17747",
       "triggerID" : "5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4212e1f283a0d86e211341b7ee4049019fe1e2c3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17746) 
   * 5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17747) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #8849: [HUDI-6304] Handle cases when inflight files are removed and job failed leaving pending rollback files

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8849:
URL: https://github.com/apache/hudi/pull/8849#discussion_r1226066053


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -704,6 +706,7 @@ && isIndexingCommit(instant.getTimestamp()))
   protected Boolean rollbackFailedWrites() {
     HoodieTable table = createTable(config, hadoopConf);
     List<String> instantsToRollback = getInstantsToRollback(table.getMetaClient(), config.getFailedWritesCleanPolicy(), Option.empty());
+    removeInflightFilesAlreadyRolledBack(instantsToRollback, table.getMetaClient());

Review Comment:
   Nice catch ~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #8849: [UBER] Rollback enhancements

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8849:
URL: https://github.com/apache/hudi/pull/8849#discussion_r1211197102


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestCopyOnWriteRollbackActionExecutor.java:
##########
@@ -216,7 +228,7 @@ public void testCopyOnWriteRollbackWithReplaceCommits(boolean isUsingMarkers) th
   }
 
   @ParameterizedTest
-  @ValueSource(booleans = {true, false})
+  @ValueSource(booleans = {true})

Review Comment:
   Why this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1569540313

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3a119ed3dfa7b2514f0603d8018a933410f08875 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504) 
   * 912f9bdc6647cc7f3df090db001b9ec84dde0f53 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [HUDI-6304][UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1586057223

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 912f9bdc6647cc7f3df090db001b9ec84dde0f53 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505) 
   * 2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] suryaprasanna commented on a diff in pull request #8849: [HUDI-6304] Handle cases when inflight files are removed and job failed leaving pending rollback files

Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #8849:
URL: https://github.com/apache/hudi/pull/8849#discussion_r1225885972


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/cluster/ClusteringTestBase.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.validator.SqlQueryEqualityPreCommitValidator;
+import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.versioning.TimelineLayoutVersion;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.FileSystemViewStorageType;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieCleanConfig;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodiePreCommitValidatorConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.testutils.HoodieClientTestUtils;
+import org.apache.hudi.testutils.MetadataMergeWriteStatus;
+
+import java.util.Properties;
+
+public class ClusteringTestBase {
+  protected static int timelineServicePort =
+      FileSystemViewStorageConfig.REMOTE_PORT_NUM.defaultValue();
+  private static final String COUNT_SQL_QUERY_FOR_VALIDATION = "select count(*) from <TABLE_NAME>";

Review Comment:
   Refactored to use Utils method instead test bases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1569494339

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3a119ed3dfa7b2514f0603d8018a933410f08875 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [HUDI-6304] Handle cases when inflight files are removed and job failed leaving pending rollback files

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1586258272

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741",
       "triggerID" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17746",
       "triggerID" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741) 
   * 4212e1f283a0d86e211341b7ee4049019fe1e2c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17746) 
   * 5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [HUDI-6304] Handle cases when inflight files are removed and job failed leaving pending rollback files

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1586328319

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741",
       "triggerID" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17746",
       "triggerID" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17747",
       "triggerID" : "5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a6b08cf64a5b7ba588f256c21d78b7bc08bd2a2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17747) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [HUDI-6304] Handle cases when inflight files are removed and job failed leaving pending rollback files

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1586256757

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741",
       "triggerID" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17746",
       "triggerID" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741) 
   * 4212e1f283a0d86e211341b7ee4049019fe1e2c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17746) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [HUDI-6304][UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1586244620

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741",
       "triggerID" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4212e1f283a0d86e211341b7ee4049019fe1e2c3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741) 
   * 4212e1f283a0d86e211341b7ee4049019fe1e2c3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] codope merged pull request #8849: [HUDI-6304] Handle cases when inflight files are removed and job failed leaving pending rollback files

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #8849:
URL: https://github.com/apache/hudi/pull/8849


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [HUDI-6304][UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1586073110

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741",
       "triggerID" : "2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2a283bf8b02466983f1eee99f9c9f58ed7f6bd0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17741) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] danny0405 commented on a diff in pull request #8849: [UBER] Rollback enhancements

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8849:
URL: https://github.com/apache/hudi/pull/8849#discussion_r1211196598


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/cluster/ClusteringTestBase.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.validator.SqlQueryEqualityPreCommitValidator;
+import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.versioning.TimelineLayoutVersion;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.FileSystemViewStorageType;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieCleanConfig;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodiePreCommitValidatorConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.testutils.HoodieClientTestUtils;
+import org.apache.hudi.testutils.MetadataMergeWriteStatus;
+
+import java.util.Properties;
+
+public class ClusteringTestBase {
+  protected static int timelineServicePort =
+      FileSystemViewStorageConfig.REMOTE_PORT_NUM.defaultValue();
+  private static final String COUNT_SQL_QUERY_FOR_VALIDATION = "select count(*) from <TABLE_NAME>";

Review Comment:
   Is there any need to extends from class `HoodieClientTestBase` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1569534183

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3a119ed3dfa7b2514f0603d8018a933410f08875 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504) 
   * 912f9bdc6647cc7f3df090db001b9ec84dde0f53 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] suryaprasanna commented on a diff in pull request #8849: [HUDI-6304] Handle cases when inflight files are removed and job failed leaving pending rollback files

Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on code in PR #8849:
URL: https://github.com/apache/hudi/pull/8849#discussion_r1225863737


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackActionExecutor.java:
##########
@@ -240,22 +244,21 @@ protected void finishRollback(HoodieInstant inflightInstant, HoodieRollbackMetad
         this.txnManager.beginTransaction(Option.of(inflightInstant), Option.empty());
       }
 
-      // If publish the rollback to the timeline, we first write the rollback metadata
-      // to metadata table
+      // If publish the rollback to the timeline, we first write the rollback metadata to metadata table
+      // Then transition the inflight rollback to completed state.
       if (!skipTimelinePublish) {
         writeTableMetadata(rollbackMetadata);
-      }
-
-      // Then we delete the inflight instant in the data table timeline if enabled
-      deleteInflightAndRequestedInstant(deleteInstants, table.getActiveTimeline(), resolvedInstant);
-
-      // If publish the rollback to the timeline, we finally transition the inflight rollback
-      // to complete in the data table timeline
-      if (!skipTimelinePublish) {
         table.getActiveTimeline().transitionRollbackInflightToComplete(inflightInstant,
             TimelineMetadataUtils.serializeRollbackMetadata(rollbackMetadata));
         LOG.info("Rollback of Commits " + rollbackMetadata.getCommitsRollback() + " is complete");
       }
+
+      // Commit to rollback instant files are deleted after the rollback commit is transitioned from inflight to completed
+      // If job were to fail after transitioning rollback from inflight to complete and before delete the instant files,
+      // then subsequent retries of the rollback for this instant will see if there is a completed rollback present for this instant
+      // and then directly delete the files and abort.

Review Comment:
   I committed the cleanup logic. Also, added test for this change.



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestCopyOnWriteRollbackActionExecutor.java:
##########
@@ -216,7 +228,7 @@ public void testCopyOnWriteRollbackWithReplaceCommits(boolean isUsingMarkers) th
   }
 
   @ParameterizedTest
-  @ValueSource(booleans = {true, false})
+  @ValueSource(booleans = {true})

Review Comment:
   Reverted this change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1569490283

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3a119ed3dfa7b2514f0603d8018a933410f08875 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] nsivabalan commented on a diff in pull request #8849: [UBER] Rollback enhancements

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8849:
URL: https://github.com/apache/hudi/pull/8849#discussion_r1212454576


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackActionExecutor.java:
##########
@@ -240,22 +244,21 @@ protected void finishRollback(HoodieInstant inflightInstant, HoodieRollbackMetad
         this.txnManager.beginTransaction(Option.of(inflightInstant), Option.empty());
       }
 
-      // If publish the rollback to the timeline, we first write the rollback metadata
-      // to metadata table
+      // If publish the rollback to the timeline, we first write the rollback metadata to metadata table
+      // Then transition the inflight rollback to completed state.
       if (!skipTimelinePublish) {
         writeTableMetadata(rollbackMetadata);
-      }
-
-      // Then we delete the inflight instant in the data table timeline if enabled
-      deleteInflightAndRequestedInstant(deleteInstants, table.getActiveTimeline(), resolvedInstant);
-
-      // If publish the rollback to the timeline, we finally transition the inflight rollback
-      // to complete in the data table timeline
-      if (!skipTimelinePublish) {
         table.getActiveTimeline().transitionRollbackInflightToComplete(inflightInstant,
             TimelineMetadataUtils.serializeRollbackMetadata(rollbackMetadata));
         LOG.info("Rollback of Commits " + rollbackMetadata.getCommitsRollback() + " is complete");
       }
+
+      // Commit to rollback instant files are deleted after the rollback commit is transitioned from inflight to completed
+      // If job were to fail after transitioning rollback from inflight to complete and before delete the instant files,
+      // then subsequent retries of the rollback for this instant will see if there is a completed rollback present for this instant
+      // and then directly delete the files and abort.

Review Comment:
   can you point me to this part of the code. Where we check for completed rollback instants and clean up the files directly ? 
   From what I know, we only check for pending rollbacks 
   ref: getPendingRollbackInfos
   and re-use the rollback instant if applicable. 
   
   Also, do we have a test for this ?



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestCopyOnWriteRollbackActionExecutor.java:
##########
@@ -328,4 +340,213 @@ private void performRollbackAndValidate(boolean isUsingMarkers, HoodieWriteConfi
 
     assertFalse(WriteMarkersFactory.get(cfg.getMarkersType(), table, commitInstant.getTimestamp()).doesMarkerDirExist());
   }
+
+  /**
+   * This method tests rollback of completed ingestion commits and replacecommit inflight files
+   * when there is another replacecommit with greater timestamp already present in the timeline.
+   */
+  @Test
+  public void testRollbackWhenReplaceCommitIsPresent() throws Exception {

Review Comment:
   can you move these into separate PRs. will help landing the PRs at diff pace. also, the PR description will be succinct. its fine for this PR. but in future



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/cluster/ClusteringTestBase.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.cluster;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.validator.SqlQueryEqualityPreCommitValidator;
+import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.fs.ConsistencyGuardConfig;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.versioning.TimelineLayoutVersion;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+import org.apache.hudi.common.table.view.FileSystemViewStorageType;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieCleanConfig;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodiePreCommitValidatorConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.testutils.HoodieClientTestUtils;
+import org.apache.hudi.testutils.MetadataMergeWriteStatus;
+
+import java.util.Properties;
+
+public class ClusteringTestBase {
+  protected static int timelineServicePort =
+      FileSystemViewStorageConfig.REMOTE_PORT_NUM.defaultValue();
+  private static final String COUNT_SQL_QUERY_FOR_VALIDATION = "select count(*) from <TABLE_NAME>";

Review Comment:
   yeah. lets try to extends from one of the existing test bases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] nsivabalan commented on pull request #8849: [UBER] Rollback enhancements

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1571156439

   can we please create a jira and link it in title.?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #8849: [UBER] Rollback enhancements

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8849:
URL: https://github.com/apache/hudi/pull/8849#issuecomment-1569946565

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17504",
       "triggerID" : "3a119ed3dfa7b2514f0603d8018a933410f08875",
       "triggerType" : "PUSH"
     }, {
       "hash" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505",
       "triggerID" : "912f9bdc6647cc7f3df090db001b9ec84dde0f53",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 912f9bdc6647cc7f3df090db001b9ec84dde0f53 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17505) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org