You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/21 02:58:19 UTC
[GitHub] [hudi] yuzhaojing opened a new pull request, #6732: [HUDI-4148] Preparations and client for hudi table management service
yuzhaojing opened a new pull request, #6732:
URL: https://github.com/apache/hudi/pull/6732
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing closed pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing closed pull request #6732: [HUDI-4148] Add client for hudi table service manager
URL: https://github.com/apache/hudi/pull/6732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1375486285
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1378222704
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190)
* e4701cd7f6ba883bca09d1c99b32135bf827c58d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058305966
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_DEPLOY_PARALLELISM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.parallelism")
+ .defaultValue(100)
+ .withDocumentation("The max parallelism to deploy for table service of this table, default 100");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXECUTION_ENGINE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".execution.engine")
+ .defaultValue("spark")
+ .withDocumentation("The execution engine to deploy for table service of this table, default spark");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.extra.params")
+ .defaultValue("")
+ .withDocumentation("The extra params to deploy for table service of this table, split by ';'");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TIMEOUT = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".timeout")
+ .defaultValue(300)
+ .withDocumentation("Connection timeout for client");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRIES = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retries")
+ .defaultValue(3)
+ .withDocumentation("Number of retries while opening a connection to table service manager");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRY_DELAY = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retry.delay")
+ .defaultValue(1)
+ .withDocumentation("Number of seconds for the client to wait between consecutive connection attempts");
+
+ public static final ConfigProperty<String> RETRY_EXCEPTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".operation.retry.exceptions")
+ .defaultValue("IOException")
+ .withDocumentation("The class name of the Exception that needs to be re-tryed, separated by commas. "
+ + "Default is empty which means retry all the IOException and RuntimeException from FileSystem");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TOLERABLE_NUM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".tolerable.num")
+ .defaultValue(0)
+ .withDocumentation("Number of connection to table manager service unsuccessful tolerable for the client");
Review Comment:
Will update this after retryHelper Improve pr merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1367108432
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1062150964
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -283,44 +270,14 @@ public void preWrite(String instantTime, WriteOperationType writeOperationType,
@Override
protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
- try (HoodieBackedTableMetadataWriter metadataWriter = initMetadataWriter()) {
- metadataWriter.update(metadata, instantTime, getHoodieTable().isTableServiceAction(actionType, instantTime));
- } catch (Exception e) {
- throw new HoodieException("Failed to update metadata", e);
- }
- }
-
- /**
- * Initialize the table metadata writer, for e.g, bootstrap the metadata table
- * from the filesystem if it does not exist.
- */
- private HoodieBackedTableMetadataWriter initMetadataWriter() {
- return (HoodieBackedTableMetadataWriter) FlinkHoodieBackedTableMetadataWriter.create(
- FlinkClientUtil.getHadoopConf(), this.config, HoodieFlinkEngineContext.DEFAULT);
+ tableServiceClient.writeTableMetadata(table, instantTime, actionType, metadata);
}
/**
* Initialized the metadata table on start up, should only be called once on driver.
*/
public void initMetadataTable() {
- HoodieFlinkTable<?> table = getHoodieTable();
- if (config.isMetadataTableEnabled()) {
- // initialize the metadata table path
- // guard the metadata writer with concurrent lock
- try {
- this.txnManager.getLockManager().lock();
- initMetadataWriter().close();
- } catch (Exception e) {
- throw new HoodieException("Failed to initialize metadata table", e);
- } finally {
- this.txnManager.getLockManager().unlock();
- }
- // clean the obsolete index stats
- table.deleteMetadataIndexIfNecessary();
- } else {
- // delete the metadata table if it was enabled but is now disabled
- table.maybeDeleteMetadataTable();
- }
+ ((HoodieFlinkTableServiceClient<T>) tableServiceClient).initMetadataTable();
Review Comment:
Ok, I will start a follow up task to normalize the initialization of the flink MDT.
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -283,44 +270,14 @@ public void preWrite(String instantTime, WriteOperationType writeOperationType,
@Override
protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
- try (HoodieBackedTableMetadataWriter metadataWriter = initMetadataWriter()) {
- metadataWriter.update(metadata, instantTime, getHoodieTable().isTableServiceAction(actionType, instantTime));
- } catch (Exception e) {
- throw new HoodieException("Failed to update metadata", e);
- }
- }
-
- /**
- * Initialize the table metadata writer, for e.g, bootstrap the metadata table
- * from the filesystem if it does not exist.
- */
- private HoodieBackedTableMetadataWriter initMetadataWriter() {
- return (HoodieBackedTableMetadataWriter) FlinkHoodieBackedTableMetadataWriter.create(
- FlinkClientUtil.getHadoopConf(), this.config, HoodieFlinkEngineContext.DEFAULT);
+ tableServiceClient.writeTableMetadata(table, instantTime, actionType, metadata);
Review Comment:
Ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1363996282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
}, {
"hash" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939",
"triggerID" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* a614791ea45854805a892bf53ec99f0dcdad30c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1364004381
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
}, {
"hash" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939",
"triggerID" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"triggerType" : "PUSH"
}, {
"hash" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13943",
"triggerID" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* a614791ea45854805a892bf53ec99f0dcdad30c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939)
* 24f7f28bb6f5d124e250afae9e5a93b51cb0bd07 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13943)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r995450411
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
+
+ public static final String EXECUTE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/execute");
+ public static final String DELETE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/delete");
+
+ public static final String EXECUTE_CLEAN = String.format("%s/%s", BASE_URL, "clean/execute");
+ public static final String DELETE_CLEAN = String.format("%s/%s", BASE_URL, "clean/delete");
+
+ public static final String DATABASE_NAME_PARAM = "db_name";
+ public static final String TABLE_NAME_PARAM = "table_name";
+ public static final String BASEPATH_PARAM = "basepath";
+ public static final String INSTANT_PARAM = "instant";
+ public static final String USERNAME = "username";
+ public static final String CLUSTER = "cluster";
+ public static final String QUEUE = "queue";
+ public static final String RESOURCE = "resource";
+ public static final String PARALLELISM = "parallelism";
+ public static final String EXTRA_PARAMS = "extra_params";
+ public static final String EXECUTION_ENGINE = "execution_engine";
+
+ private final HoodieTableManagerConfig config;
+ private final HoodieTableMetaClient metaClient;
+ private final String host;
+ private final int port;
+ private final String basePath;
+ private final String dbName;
+ private final String tableName;
+
+ private static final Logger LOG = LogManager.getLogger(HoodieTableManagerClient.class);
+
+ public HoodieTableManagerClient(HoodieTableMetaClient metaClient, HoodieTableManagerConfig config) {
+ this.basePath = metaClient.getBasePathV2().toString();
+ this.dbName = metaClient.getTableConfig().getDatabaseName();
+ this.tableName = metaClient.getTableConfig().getTableName();
+ this.host = config.getTableManagerHost();
+ this.port = config.getTableManagerPort();
+ this.config = config;
+ this.metaClient = metaClient;
+ }
+
+ private String executeRequest(String requestPath, Map<String, String> queryParameters) throws IOException {
+ URIBuilder builder =
+ new URIBuilder().setHost(host).setPort(port).setPath(requestPath).setScheme("http");
+ queryParameters.forEach(builder::addParameter);
+
+ String url = builder.toString();
+ LOG.info("Sending request to table management service : (" + url + ")");
+ Response response;
+ int timeout = this.config.getConnectionTimeout() * 1000; // msec
+ int requestRetryLimit = config.getConnectionRetryLimit();
+ int retry = 0;
+
+ while (retry < requestRetryLimit) {
+ try {
+ response = Request.Get(url).connectTimeout(timeout).socketTimeout(timeout).execute();
+ return response.returnContent().asString();
+ } catch (IOException e) {
+ retry++;
+ LOG.warn(String.format("Failed request to server %s, will retry for %d times", url, requestRetryLimit - retry), e);
+ if (requestRetryLimit == retry) {
+ throw e;
+ }
+ }
+
+ try {
+ TimeUnit.SECONDS.sleep(config.getConnectionRetryDelay());
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ }
+
+ throw new IOException(String.format("Failed request to table management service %s after retry %d times", url, requestRetryLimit));
+ }
+
+ private Map<String, String> getParamsWithAdditionalParams(String[] paramNames, String[] paramVals) {
Review Comment:
This method name aligned with RemoteHoodieTableFileSystemView
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1307652223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 36e29156d84985f88e46463b6177e3e751bbe013 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1297026200
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689)
* 7cf07c0681d32e8831af9f39168695d5540af636 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r995432589
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
+
+ public static final String EXECUTE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/execute");
+ public static final String DELETE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/delete");
+
+ public static final String EXECUTE_CLEAN = String.format("%s/%s", BASE_URL, "clean/execute");
+ public static final String DELETE_CLEAN = String.format("%s/%s", BASE_URL, "clean/delete");
+
+ public static final String DATABASE_NAME_PARAM = "db_name";
+ public static final String TABLE_NAME_PARAM = "table_name";
+ public static final String BASEPATH_PARAM = "basepath";
+ public static final String INSTANT_PARAM = "instant";
+ public static final String USERNAME = "username";
+ public static final String CLUSTER = "cluster";
+ public static final String QUEUE = "queue";
+ public static final String RESOURCE = "resource";
+ public static final String PARALLELISM = "parallelism";
+ public static final String EXTRA_PARAMS = "extra_params";
+ public static final String EXECUTION_ENGINE = "execution_engine";
+
+ private final HoodieTableManagerConfig config;
+ private final HoodieTableMetaClient metaClient;
+ private final String host;
+ private final int port;
+ private final String basePath;
+ private final String dbName;
+ private final String tableName;
+
+ private static final Logger LOG = LogManager.getLogger(HoodieTableManagerClient.class);
+
+ public HoodieTableManagerClient(HoodieTableMetaClient metaClient, HoodieTableManagerConfig config) {
+ this.basePath = metaClient.getBasePathV2().toString();
+ this.dbName = metaClient.getTableConfig().getDatabaseName();
+ this.tableName = metaClient.getTableConfig().getTableName();
+ this.host = config.getTableManagerHost();
+ this.port = config.getTableManagerPort();
+ this.config = config;
+ this.metaClient = metaClient;
+ }
+
+ private String executeRequest(String requestPath, Map<String, String> queryParameters) throws IOException {
+ URIBuilder builder =
+ new URIBuilder().setHost(host).setPort(port).setPath(requestPath).setScheme("http");
+ queryParameters.forEach(builder::addParameter);
+
+ String url = builder.toString();
+ LOG.info("Sending request to table management service : (" + url + ")");
+ Response response;
+ int timeout = this.config.getConnectionTimeout() * 1000; // msec
+ int requestRetryLimit = config.getConnectionRetryLimit();
+ int retry = 0;
+
+ while (retry < requestRetryLimit) {
Review Comment:
Already update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1278908012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* d4408906bcc13c55f6350f35c168be32d978daf6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253398407
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 149e8f3c61e1221c4659da00a4168faeed1174dd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547)
* 78f3167ee6aba247ef55dcfacf8c2be603d621be Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1363566229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* f7e140b4aff4e87ca18e8390faa04e1c72b92378 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366647277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64ecea100e226b7fd539cab05c03bc9902e36db1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1367133264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* c20aa589730546c0c7bb82969c92aa6d364af101 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366360106
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374042226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374481061
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374724389
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366338370
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd837f2febbd7c102305f91f8e85532782d29467 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366362106
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058485652
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -277,7 +277,7 @@ protected void writeTableMetadata(HoodieTable table, String instantTime, String
* Initialized the metadata table on start up, should only be called once on driver.
*/
public void initMetadataTable() {
- ((HoodieFlinkTableServiceClient<T>) tableServiceClient).initMetadataTable();
+ ((HoodieFlinkHoodieTableServiceClient<T>) tableServiceClient).initMetadataTable();
Review Comment:
the diagram needs update too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058482453
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -277,7 +277,7 @@ protected void writeTableMetadata(HoodieTable table, String instantTime, String
* Initialized the metadata table on start up, should only be called once on driver.
*/
public void initMetadataTable() {
- ((HoodieFlinkTableServiceClient<T>) tableServiceClient).initMetadataTable();
+ ((HoodieFlinkHoodieTableServiceClient<T>) tableServiceClient).initMetadataTable();
Review Comment:
redundant `Hoodie` in the name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1362989435
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 27ae850689d655b0054d4a106c10b418ff662fda Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949)
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* f7e140b4aff4e87ca18e8390faa04e1c72b92378 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #6732:
URL: https://github.com/apache/hudi/pull/6732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1306532895
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 0da5a658baa0616cfb0666fe6a1d0d916191d274 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1305702180
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 7cf07c0681d32e8831af9f39168695d5540af636 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850)
* 0da5a658baa0616cfb0666fe6a1d0d916191d274 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1039631891
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/CommonHoodieClient.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.embedded.EmbeddedTimelineService;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.client.transaction.TransactionManager;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class CommonHoodieClient extends BaseHoodieClient {
+
+ private static final Logger LOG = LogManager.getLogger(CommonHoodieClient.class);
+
+ protected final transient HoodieMetrics metrics;
+ protected transient Timer.Context writeTimer = null;
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient WriteOperationType operationType;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+ protected final TransactionManager txnManager;
+
+ protected CommonHoodieClient(HoodieEngineContext context, HoodieWriteConfig clientConfig, Option<EmbeddedTimelineService> timelineServer) {
+ super(context, clientConfig, timelineServer);
+ this.metrics = new HoodieMetrics(config);
+ this.txnManager = new TransactionManager(config, fs);
+ }
+
+ public void setOperationType(WriteOperationType operationType) {
+ this.operationType = operationType;
+ }
+
+ public WriteOperationType getOperationType() {
+ return this.operationType;
+ }
+
+ /**
+ * Get inflight time line exclude compaction and clustering.
+ * @param metaClient
+ * @return
+ */
+ private HoodieTimeline getInflightTimelineExcludeCompactionAndClustering(HoodieTableMetaClient metaClient) {
+ HoodieTimeline inflightTimelineWithReplaceCommit = metaClient.getCommitsTimeline().filterPendingExcludingCompaction();
+ HoodieTimeline inflightTimelineExcludeClusteringCommit = inflightTimelineWithReplaceCommit.filter(instant -> {
+ if (instant.getAction().equals(HoodieTimeline.REPLACE_COMMIT_ACTION)) {
+ Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(metaClient, instant);
+ return !instantPlan.isPresent();
+ } else {
+ return true;
+ }
+ });
+ return inflightTimelineExcludeClusteringCommit;
+ }
+
+ protected Option<HoodiePendingRollbackInfo> getPendingRollbackInfo(HoodieTableMetaClient metaClient, String commitToRollback) {
+ return getPendingRollbackInfo(metaClient, commitToRollback, true);
+ }
+
+ public Option<HoodiePendingRollbackInfo> getPendingRollbackInfo(HoodieTableMetaClient metaClient, String commitToRollback, boolean ignoreCompactionAndClusteringInstants) {
+ return getPendingRollbackInfos(metaClient, ignoreCompactionAndClusteringInstants).getOrDefault(commitToRollback, Option.empty());
+ }
+
+ protected Map<String, Option<HoodiePendingRollbackInfo>> getPendingRollbackInfos(HoodieTableMetaClient metaClient) {
+ return getPendingRollbackInfos(metaClient, true);
+ }
+
+ /**
+ * Fetch map of pending commits to be rolled-back to {@link HoodiePendingRollbackInfo}.
+ * @param metaClient instance of {@link HoodieTableMetaClient} to use.
+ * @return map of pending commits to be rolled-back instants to Rollback Instant and Rollback plan Pair.
+ */
+ protected Map<String, Option<HoodiePendingRollbackInfo>> getPendingRollbackInfos(HoodieTableMetaClient metaClient, boolean ignoreCompactionAndClusteringInstants) {
+ List<HoodieInstant> instants = metaClient.getActiveTimeline().filterPendingRollbackTimeline().getInstants().collect(Collectors.toList());
+ Map<String, Option<HoodiePendingRollbackInfo>> infoMap = new HashMap<>();
+ for (HoodieInstant rollbackInstant : instants) {
+ HoodieRollbackPlan rollbackPlan;
+ try {
+ rollbackPlan = RollbackUtils.getRollbackPlan(metaClient, rollbackInstant);
+ } catch (Exception e) {
+ if (rollbackInstant.isRequested()) {
+ LOG.warn("Fetching rollback plan failed for " + rollbackInstant + ", deleting the plan since it's in REQUESTED state", e);
+ try {
+ metaClient.getActiveTimeline().deletePending(rollbackInstant);
+ } catch (HoodieIOException he) {
+ LOG.warn("Cannot delete " + rollbackInstant, he);
+ continue;
+ }
+ } else {
+ // Here we assume that if the rollback is inflight, the rollback plan is intact
+ // in instant.rollback.requested. The exception here can be due to other reasons.
+ LOG.warn("Fetching rollback plan failed for " + rollbackInstant + ", skip the plan", e);
+ }
+ continue;
+ }
+
+ try {
+ String action = rollbackPlan.getInstantToRollback().getAction();
+ if (ignoreCompactionAndClusteringInstants) {
+ if (!HoodieTimeline.COMPACTION_ACTION.equals(action)) {
+ boolean isClustering = HoodieTimeline.REPLACE_COMMIT_ACTION.equals(action)
+ && ClusteringUtils.getClusteringPlan(metaClient, new HoodieInstant(true, rollbackPlan.getInstantToRollback().getAction(),
+ rollbackPlan.getInstantToRollback().getCommitTime())).isPresent();
+ if (!isClustering) {
+ String instantToRollback = rollbackPlan.getInstantToRollback().getCommitTime();
+ infoMap.putIfAbsent(instantToRollback, Option.of(new HoodiePendingRollbackInfo(rollbackInstant, rollbackPlan)));
+ }
+ }
+ } else {
+ infoMap.putIfAbsent(rollbackPlan.getInstantToRollback().getCommitTime(), Option.of(new HoodiePendingRollbackInfo(rollbackInstant, rollbackPlan)));
+ }
+ } catch (Exception e) {
+ LOG.warn("Processing rollback plan failed for " + rollbackInstant + ", skip the plan", e);
+ }
+ }
+ return infoMap;
+ }
+
+ /**
+ * Rollback all failed writes.
+ */
+ protected Boolean rollbackFailedWrites() {
+ return rollbackFailedWrites(false);
+ }
+
+ /**
+ * Rollback all failed writes.
+ * @param skipLocking if this is triggered by another parent transaction, locking can be skipped.
+ */
+ protected Boolean rollbackFailedWrites(boolean skipLocking) {
+ HoodieTable table = createTable(config, hadoopConf);
+ List<String> instantsToRollback = getInstantsToRollback(table.getMetaClient(), config.getFailedWritesCleanPolicy(), Option.empty());
+ Map<String, Option<HoodiePendingRollbackInfo>> pendingRollbacks = getPendingRollbackInfos(table.getMetaClient());
+ instantsToRollback.forEach(entry -> pendingRollbacks.putIfAbsent(entry, Option.empty()));
+ rollbackFailedWrites(pendingRollbacks, skipLocking);
+ return true;
+ }
+
+ protected void rollbackFailedWrites(Map<String, Option<HoodiePendingRollbackInfo>> instantsToRollback, boolean skipLocking) {
+ // sort in reverse order of commit times
+ LinkedHashMap<String, Option<HoodiePendingRollbackInfo>> reverseSortedRollbackInstants = instantsToRollback.entrySet()
+ .stream().sorted((i1, i2) -> i2.getKey().compareTo(i1.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue, (e1, e2) -> e1, LinkedHashMap::new));
+ for (Map.Entry<String, Option<HoodiePendingRollbackInfo>> entry : reverseSortedRollbackInstants.entrySet()) {
+ if (HoodieTimeline.compareTimestamps(entry.getKey(), HoodieTimeline.LESSER_THAN_OR_EQUALS,
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS)) {
+ // do we need to handle failed rollback of a bootstrap
+ rollbackFailedBootstrap();
+ HeartbeatUtils.deleteHeartbeatFile(fs, basePath, entry.getKey(), config);
+ break;
+ } else {
+ rollback(entry.getKey(), entry.getValue(), skipLocking);
+ HeartbeatUtils.deleteHeartbeatFile(fs, basePath, entry.getKey(), config);
+ }
+ }
+ }
+
+ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, HoodieFailedWritesCleaningPolicy cleaningPolicy, Option<String> curInstantTime) {
+ Stream<HoodieInstant> inflightInstantsStream = getInflightTimelineExcludeCompactionAndClustering(metaClient)
+ .getReverseOrderedInstants();
+ if (cleaningPolicy.isEager()) {
+ return inflightInstantsStream.map(HoodieInstant::getTimestamp).filter(entry -> {
+ if (curInstantTime.isPresent()) {
+ return !entry.equals(curInstantTime.get());
+ } else {
+ return true;
+ }
+ }).collect(Collectors.toList());
+ } else if (cleaningPolicy.isLazy()) {
+ return inflightInstantsStream.filter(instant -> {
+ try {
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ } else if (cleaningPolicy.isNever()) {
+ return Collections.EMPTY_LIST;
+ } else {
+ throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
+ }
+ }
+
+ /**
+ * @Deprecated
+ * Rollback the inflight record changes with the given commit time. This
+ * will be removed in future in favor of {@link BaseHoodieWriteClient#restoreToInstant(String, boolean)
+ *
+ * @param commitInstantTime Instant time of the commit
+ * @param pendingRollbackInfo pending rollback instant and plan if rollback failed from previous attempt.
+ * @param skipLocking if this is triggered by another parent transaction, locking can be skipped.
+ * @throws HoodieRollbackException if rollback cannot be performed successfully
+ */
+ @Deprecated
+ public boolean rollback(final String commitInstantTime, Option<HoodiePendingRollbackInfo> pendingRollbackInfo, boolean skipLocking) throws HoodieRollbackException {
+ LOG.info("Begin rollback of instant " + commitInstantTime);
+ final String rollbackInstantTime = pendingRollbackInfo.map(entry -> entry.getRollbackInstant().getTimestamp()).orElse(HoodieActiveTimeline.createNewInstantTime());
+ final Timer.Context timerContext = this.metrics.getRollbackCtx();
+ try {
+ HoodieTable table = createTable(config, hadoopConf);
+ Option<HoodieInstant> commitInstantOpt = Option.fromJavaOptional(table.getActiveTimeline().getCommitsTimeline().getInstants()
+ .filter(instant -> HoodieActiveTimeline.EQUALS.test(instant.getTimestamp(), commitInstantTime))
+ .findFirst());
+ if (commitInstantOpt.isPresent() || pendingRollbackInfo.isPresent()) {
+ LOG.info(String.format("Scheduling Rollback at instant time : %s "
+ + "(exists in active timeline: %s), with rollback plan: %s",
+ rollbackInstantTime, commitInstantOpt.isPresent(), pendingRollbackInfo.isPresent()));
+ Option<HoodieRollbackPlan> rollbackPlanOption = pendingRollbackInfo.map(entry -> Option.of(entry.getRollbackPlan()))
+ .orElseGet(() -> table.scheduleRollback(context, rollbackInstantTime, commitInstantOpt.get(), false, config.shouldRollbackUsingMarkers()));
+ if (rollbackPlanOption.isPresent()) {
+ // There can be a case where the inflight rollback failed after the instant files
+ // are deleted for commitInstantTime, so that commitInstantOpt is empty as it is
+ // not present in the timeline. In such a case, the hoodie instant instance
+ // is reconstructed to allow the rollback to be reattempted, and the deleteInstants
+ // is set to false since they are already deleted.
+ // Execute rollback
+ HoodieRollbackMetadata rollbackMetadata = commitInstantOpt.isPresent()
+ ? table.rollback(context, rollbackInstantTime, commitInstantOpt.get(), true, skipLocking)
+ : table.rollback(context, rollbackInstantTime, new HoodieInstant(
+ true, rollbackPlanOption.get().getInstantToRollback().getAction(), commitInstantTime),
+ false, skipLocking);
+ if (timerContext != null) {
+ long durationInMs = metrics.getDurationInMs(timerContext.stop());
+ metrics.updateRollbackMetrics(durationInMs, rollbackMetadata.getTotalFilesDeleted());
+ }
+ return true;
+ } else {
+ throw new HoodieRollbackException("Failed to rollback " + config.getBasePath() + " commits " + commitInstantTime);
+ }
+ } else {
+ LOG.warn("Cannot find instant " + commitInstantTime + " in the timeline, for rollback");
+ return false;
+ }
+ } catch (Exception e) {
+ throw new HoodieRollbackException("Failed to rollback " + config.getBasePath() + " commits " + commitInstantTime, e);
+ }
+ }
+
+ /**
+ * Main API to rollback failed bootstrap.
+ */
+ protected void rollbackFailedBootstrap() {
+ LOG.info("Rolling back pending bootstrap if present");
+ HoodieTable table = createTable(config, hadoopConf);
+ HoodieTimeline inflightTimeline = table.getMetaClient().getCommitsTimeline().filterPendingExcludingMajorAndMinorCompaction();
+ Option<String> instant = Option.fromJavaOptional(
+ inflightTimeline.getReverseOrderedInstants().map(HoodieInstant::getTimestamp).findFirst());
+ if (instant.isPresent() && HoodieTimeline.compareTimestamps(instant.get(), HoodieTimeline.LESSER_THAN_OR_EQUALS,
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS)) {
+ LOG.info("Found pending bootstrap instants. Rolling them back");
+ table.rollbackBootstrap(context, HoodieActiveTimeline.createNewInstantTime());
+ LOG.info("Finished rolling back pending bootstrap");
+ }
+ }
+
+ protected abstract HoodieTable createTable(HoodieWriteConfig config, Configuration hadoopConf);
+
+ @Override
+ public void close() {
+ AsyncArchiveService.forceShutdown(asyncArchiveService);
+ asyncArchiveService = null;
+ AsyncCleanerService.forceShutdown(asyncCleanerService);
+ asyncCleanerService = null;
Review Comment:
we should keep the closing of table services under TableServiceClient. To stay compatible for write clients, we can delegate the closing to some table service client. so we can keep the responsibilities clearly separated
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/CommonHoodieClient.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.embedded.EmbeddedTimelineService;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.client.transaction.TransactionManager;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class CommonHoodieClient extends BaseHoodieClient {
+
+ private static final Logger LOG = LogManager.getLogger(CommonHoodieClient.class);
+
+ protected final transient HoodieMetrics metrics;
+ protected transient Timer.Context writeTimer = null;
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient WriteOperationType operationType;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+ protected final TransactionManager txnManager;
+
+ protected CommonHoodieClient(HoodieEngineContext context, HoodieWriteConfig clientConfig, Option<EmbeddedTimelineService> timelineServer) {
+ super(context, clientConfig, timelineServer);
+ this.metrics = new HoodieMetrics(config);
+ this.txnManager = new TransactionManager(config, fs);
+ }
+
+ public void setOperationType(WriteOperationType operationType) {
+ this.operationType = operationType;
+ }
+
+ public WriteOperationType getOperationType() {
+ return this.operationType;
+ }
+
+ /**
+ * Get inflight time line exclude compaction and clustering.
+ * @param metaClient
+ * @return
+ */
+ private HoodieTimeline getInflightTimelineExcludeCompactionAndClustering(HoodieTableMetaClient metaClient) {
Review Comment:
this does not seem to fit in a client class.. can it be static and moved to some timeline related class?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseTableServiceClient.java:
##########
@@ -0,0 +1,519 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+
+public abstract class BaseTableServiceClient<O> extends CommonHoodieClient
Review Comment:
CommonHoodieClient is pretty much all about rollback, which is also a table service. Understand that write client should also perform rollbacks, so can we keep rollback under BaseTableServiceClient and let write client delegate it to an instance of TableServiceClient ? then we can remove CommonHoodieClient to simplify the hierarchy.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/CommonHoodieClient.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.embedded.EmbeddedTimelineService;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.client.transaction.TransactionManager;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class CommonHoodieClient extends BaseHoodieClient {
+
+ private static final Logger LOG = LogManager.getLogger(CommonHoodieClient.class);
+
+ protected final transient HoodieMetrics metrics;
+ protected transient Timer.Context writeTimer = null;
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient WriteOperationType operationType;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+ protected final TransactionManager txnManager;
+
+ protected CommonHoodieClient(HoodieEngineContext context, HoodieWriteConfig clientConfig, Option<EmbeddedTimelineService> timelineServer) {
+ super(context, clientConfig, timelineServer);
+ this.metrics = new HoodieMetrics(config);
+ this.txnManager = new TransactionManager(config, fs);
+ }
+
+ public void setOperationType(WriteOperationType operationType) {
+ this.operationType = operationType;
+ }
+
+ public WriteOperationType getOperationType() {
+ return this.operationType;
+ }
+
+ /**
+ * Get inflight time line exclude compaction and clustering.
+ * @param metaClient
+ * @return
+ */
+ private HoodieTimeline getInflightTimelineExcludeCompactionAndClustering(HoodieTableMetaClient metaClient) {
+ HoodieTimeline inflightTimelineWithReplaceCommit = metaClient.getCommitsTimeline().filterPendingExcludingCompaction();
+ HoodieTimeline inflightTimelineExcludeClusteringCommit = inflightTimelineWithReplaceCommit.filter(instant -> {
+ if (instant.getAction().equals(HoodieTimeline.REPLACE_COMMIT_ACTION)) {
+ Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(metaClient, instant);
+ return !instantPlan.isPresent();
+ } else {
+ return true;
+ }
+ });
+ return inflightTimelineExcludeClusteringCommit;
+ }
+
+ protected Option<HoodiePendingRollbackInfo> getPendingRollbackInfo(HoodieTableMetaClient metaClient, String commitToRollback) {
+ return getPendingRollbackInfo(metaClient, commitToRollback, true);
+ }
+
+ public Option<HoodiePendingRollbackInfo> getPendingRollbackInfo(HoodieTableMetaClient metaClient, String commitToRollback, boolean ignoreCompactionAndClusteringInstants) {
+ return getPendingRollbackInfos(metaClient, ignoreCompactionAndClusteringInstants).getOrDefault(commitToRollback, Option.empty());
+ }
+
+ protected Map<String, Option<HoodiePendingRollbackInfo>> getPendingRollbackInfos(HoodieTableMetaClient metaClient) {
+ return getPendingRollbackInfos(metaClient, true);
+ }
+
+ /**
+ * Fetch map of pending commits to be rolled-back to {@link HoodiePendingRollbackInfo}.
+ * @param metaClient instance of {@link HoodieTableMetaClient} to use.
+ * @return map of pending commits to be rolled-back instants to Rollback Instant and Rollback plan Pair.
+ */
+ protected Map<String, Option<HoodiePendingRollbackInfo>> getPendingRollbackInfos(HoodieTableMetaClient metaClient, boolean ignoreCompactionAndClusteringInstants) {
+ List<HoodieInstant> instants = metaClient.getActiveTimeline().filterPendingRollbackTimeline().getInstants().collect(Collectors.toList());
+ Map<String, Option<HoodiePendingRollbackInfo>> infoMap = new HashMap<>();
+ for (HoodieInstant rollbackInstant : instants) {
+ HoodieRollbackPlan rollbackPlan;
+ try {
+ rollbackPlan = RollbackUtils.getRollbackPlan(metaClient, rollbackInstant);
+ } catch (Exception e) {
+ if (rollbackInstant.isRequested()) {
+ LOG.warn("Fetching rollback plan failed for " + rollbackInstant + ", deleting the plan since it's in REQUESTED state", e);
+ try {
+ metaClient.getActiveTimeline().deletePending(rollbackInstant);
+ } catch (HoodieIOException he) {
+ LOG.warn("Cannot delete " + rollbackInstant, he);
+ continue;
+ }
+ } else {
+ // Here we assume that if the rollback is inflight, the rollback plan is intact
+ // in instant.rollback.requested. The exception here can be due to other reasons.
+ LOG.warn("Fetching rollback plan failed for " + rollbackInstant + ", skip the plan", e);
+ }
+ continue;
+ }
+
+ try {
+ String action = rollbackPlan.getInstantToRollback().getAction();
+ if (ignoreCompactionAndClusteringInstants) {
+ if (!HoodieTimeline.COMPACTION_ACTION.equals(action)) {
+ boolean isClustering = HoodieTimeline.REPLACE_COMMIT_ACTION.equals(action)
+ && ClusteringUtils.getClusteringPlan(metaClient, new HoodieInstant(true, rollbackPlan.getInstantToRollback().getAction(),
+ rollbackPlan.getInstantToRollback().getCommitTime())).isPresent();
+ if (!isClustering) {
+ String instantToRollback = rollbackPlan.getInstantToRollback().getCommitTime();
+ infoMap.putIfAbsent(instantToRollback, Option.of(new HoodiePendingRollbackInfo(rollbackInstant, rollbackPlan)));
+ }
+ }
+ } else {
+ infoMap.putIfAbsent(rollbackPlan.getInstantToRollback().getCommitTime(), Option.of(new HoodiePendingRollbackInfo(rollbackInstant, rollbackPlan)));
+ }
+ } catch (Exception e) {
+ LOG.warn("Processing rollback plan failed for " + rollbackInstant + ", skip the plan", e);
+ }
+ }
+ return infoMap;
+ }
+
+ /**
+ * Rollback all failed writes.
+ */
+ protected Boolean rollbackFailedWrites() {
+ return rollbackFailedWrites(false);
+ }
+
+ /**
+ * Rollback all failed writes.
+ * @param skipLocking if this is triggered by another parent transaction, locking can be skipped.
+ */
+ protected Boolean rollbackFailedWrites(boolean skipLocking) {
+ HoodieTable table = createTable(config, hadoopConf);
+ List<String> instantsToRollback = getInstantsToRollback(table.getMetaClient(), config.getFailedWritesCleanPolicy(), Option.empty());
+ Map<String, Option<HoodiePendingRollbackInfo>> pendingRollbacks = getPendingRollbackInfos(table.getMetaClient());
+ instantsToRollback.forEach(entry -> pendingRollbacks.putIfAbsent(entry, Option.empty()));
+ rollbackFailedWrites(pendingRollbacks, skipLocking);
+ return true;
+ }
+
+ protected void rollbackFailedWrites(Map<String, Option<HoodiePendingRollbackInfo>> instantsToRollback, boolean skipLocking) {
+ // sort in reverse order of commit times
+ LinkedHashMap<String, Option<HoodiePendingRollbackInfo>> reverseSortedRollbackInstants = instantsToRollback.entrySet()
+ .stream().sorted((i1, i2) -> i2.getKey().compareTo(i1.getKey()))
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue, (e1, e2) -> e1, LinkedHashMap::new));
+ for (Map.Entry<String, Option<HoodiePendingRollbackInfo>> entry : reverseSortedRollbackInstants.entrySet()) {
+ if (HoodieTimeline.compareTimestamps(entry.getKey(), HoodieTimeline.LESSER_THAN_OR_EQUALS,
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS)) {
+ // do we need to handle failed rollback of a bootstrap
+ rollbackFailedBootstrap();
+ HeartbeatUtils.deleteHeartbeatFile(fs, basePath, entry.getKey(), config);
+ break;
+ } else {
+ rollback(entry.getKey(), entry.getValue(), skipLocking);
+ HeartbeatUtils.deleteHeartbeatFile(fs, basePath, entry.getKey(), config);
+ }
+ }
+ }
+
+ protected List<String> getInstantsToRollback(HoodieTableMetaClient metaClient, HoodieFailedWritesCleaningPolicy cleaningPolicy, Option<String> curInstantTime) {
+ Stream<HoodieInstant> inflightInstantsStream = getInflightTimelineExcludeCompactionAndClustering(metaClient)
+ .getReverseOrderedInstants();
+ if (cleaningPolicy.isEager()) {
+ return inflightInstantsStream.map(HoodieInstant::getTimestamp).filter(entry -> {
+ if (curInstantTime.isPresent()) {
+ return !entry.equals(curInstantTime.get());
+ } else {
+ return true;
+ }
+ }).collect(Collectors.toList());
+ } else if (cleaningPolicy.isLazy()) {
+ return inflightInstantsStream.filter(instant -> {
+ try {
+ return heartbeatClient.isHeartbeatExpired(instant.getTimestamp());
+ } catch (IOException io) {
+ throw new HoodieException("Failed to check heartbeat for instant " + instant, io);
+ }
+ }).map(HoodieInstant::getTimestamp).collect(Collectors.toList());
+ } else if (cleaningPolicy.isNever()) {
+ return Collections.EMPTY_LIST;
+ } else {
+ throw new IllegalArgumentException("Invalid Failed Writes Cleaning Policy " + config.getFailedWritesCleanPolicy());
+ }
+ }
+
+ /**
+ * @Deprecated
+ * Rollback the inflight record changes with the given commit time. This
+ * will be removed in future in favor of {@link BaseHoodieWriteClient#restoreToInstant(String, boolean)
+ *
+ * @param commitInstantTime Instant time of the commit
+ * @param pendingRollbackInfo pending rollback instant and plan if rollback failed from previous attempt.
+ * @param skipLocking if this is triggered by another parent transaction, locking can be skipped.
+ * @throws HoodieRollbackException if rollback cannot be performed successfully
+ */
+ @Deprecated
+ public boolean rollback(final String commitInstantTime, Option<HoodiePendingRollbackInfo> pendingRollbackInfo, boolean skipLocking) throws HoodieRollbackException {
+ LOG.info("Begin rollback of instant " + commitInstantTime);
+ final String rollbackInstantTime = pendingRollbackInfo.map(entry -> entry.getRollbackInstant().getTimestamp()).orElse(HoodieActiveTimeline.createNewInstantTime());
+ final Timer.Context timerContext = this.metrics.getRollbackCtx();
+ try {
+ HoodieTable table = createTable(config, hadoopConf);
+ Option<HoodieInstant> commitInstantOpt = Option.fromJavaOptional(table.getActiveTimeline().getCommitsTimeline().getInstants()
+ .filter(instant -> HoodieActiveTimeline.EQUALS.test(instant.getTimestamp(), commitInstantTime))
+ .findFirst());
+ if (commitInstantOpt.isPresent() || pendingRollbackInfo.isPresent()) {
+ LOG.info(String.format("Scheduling Rollback at instant time : %s "
+ + "(exists in active timeline: %s), with rollback plan: %s",
+ rollbackInstantTime, commitInstantOpt.isPresent(), pendingRollbackInfo.isPresent()));
+ Option<HoodieRollbackPlan> rollbackPlanOption = pendingRollbackInfo.map(entry -> Option.of(entry.getRollbackPlan()))
+ .orElseGet(() -> table.scheduleRollback(context, rollbackInstantTime, commitInstantOpt.get(), false, config.shouldRollbackUsingMarkers()));
+ if (rollbackPlanOption.isPresent()) {
+ // There can be a case where the inflight rollback failed after the instant files
+ // are deleted for commitInstantTime, so that commitInstantOpt is empty as it is
+ // not present in the timeline. In such a case, the hoodie instant instance
+ // is reconstructed to allow the rollback to be reattempted, and the deleteInstants
+ // is set to false since they are already deleted.
+ // Execute rollback
+ HoodieRollbackMetadata rollbackMetadata = commitInstantOpt.isPresent()
+ ? table.rollback(context, rollbackInstantTime, commitInstantOpt.get(), true, skipLocking)
+ : table.rollback(context, rollbackInstantTime, new HoodieInstant(
+ true, rollbackPlanOption.get().getInstantToRollback().getAction(), commitInstantTime),
+ false, skipLocking);
+ if (timerContext != null) {
+ long durationInMs = metrics.getDurationInMs(timerContext.stop());
+ metrics.updateRollbackMetrics(durationInMs, rollbackMetadata.getTotalFilesDeleted());
+ }
+ return true;
+ } else {
+ throw new HoodieRollbackException("Failed to rollback " + config.getBasePath() + " commits " + commitInstantTime);
+ }
+ } else {
+ LOG.warn("Cannot find instant " + commitInstantTime + " in the timeline, for rollback");
+ return false;
+ }
+ } catch (Exception e) {
+ throw new HoodieRollbackException("Failed to rollback " + config.getBasePath() + " commits " + commitInstantTime, e);
+ }
+ }
+
+ /**
+ * Main API to rollback failed bootstrap.
+ */
+ protected void rollbackFailedBootstrap() {
+ LOG.info("Rolling back pending bootstrap if present");
+ HoodieTable table = createTable(config, hadoopConf);
+ HoodieTimeline inflightTimeline = table.getMetaClient().getCommitsTimeline().filterPendingExcludingMajorAndMinorCompaction();
+ Option<String> instant = Option.fromJavaOptional(
+ inflightTimeline.getReverseOrderedInstants().map(HoodieInstant::getTimestamp).findFirst());
+ if (instant.isPresent() && HoodieTimeline.compareTimestamps(instant.get(), HoodieTimeline.LESSER_THAN_OR_EQUALS,
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS)) {
+ LOG.info("Found pending bootstrap instants. Rolling them back");
+ table.rollbackBootstrap(context, HoodieActiveTimeline.createNewInstantTime());
+ LOG.info("Finished rolling back pending bootstrap");
+ }
+ }
+
+ protected abstract HoodieTable createTable(HoodieWriteConfig config, Configuration hadoopConf);
Review Comment:
should this only be under BaseWriteClient? as table service client should not be responsible for creating a table.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTableManagementClient.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.common.config.HoodieTableManagementConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.RetryHelper;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagementClient {
Review Comment:
let's have a naming discussion offline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1296939708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* d4408906bcc13c55f6350f35c168be32d978daf6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204)
* df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058305796
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_DEPLOY_PARALLELISM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.parallelism")
+ .defaultValue(100)
+ .withDocumentation("The max parallelism to deploy for table service of this table, default 100");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXECUTION_ENGINE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".execution.engine")
+ .defaultValue("spark")
+ .withDocumentation("The execution engine to deploy for table service of this table, default spark");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.extra.params")
+ .defaultValue("")
+ .withDocumentation("The extra params to deploy for table service of this table, split by ';'");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TIMEOUT = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".timeout")
+ .defaultValue(300)
+ .withDocumentation("Connection timeout for client");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRIES = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retries")
+ .defaultValue(3)
+ .withDocumentation("Number of retries while opening a connection to table service manager");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRY_DELAY = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retry.delay")
+ .defaultValue(1)
+ .withDocumentation("Number of seconds for the client to wait between consecutive connection attempts");
+
+ public static final ConfigProperty<String> RETRY_EXCEPTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".operation.retry.exceptions")
+ .defaultValue("IOException")
Review Comment:
Will update this after retryHelper Improve pr merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1375489658
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1372189427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114",
"triggerID" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"triggerType" : "PUSH"
}, {
"hash" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14121",
"triggerID" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28aacdc92d0f7027e03cb2cd8891b19b48e47da0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114)
* dff9924f25bf457dc12cdd7a7a11b923ad08c3b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14121)
* af728e3eeb1fd694eba037bc9a48869831ddb053 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1277298754
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78f3167ee6aba247ef55dcfacf8c2be603d621be Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549)
* b35c4e70a6564f5524b1e662142b51d419ccaeff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1277305680
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78f3167ee6aba247ef55dcfacf8c2be603d621be Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549)
* b35c4e70a6564f5524b1e662142b51d419ccaeff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r995438832
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
Review Comment:
Already update.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
+
+ public static final String EXECUTE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/execute");
+ public static final String DELETE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/delete");
+
+ public static final String EXECUTE_CLEAN = String.format("%s/%s", BASE_URL, "clean/execute");
+ public static final String DELETE_CLEAN = String.format("%s/%s", BASE_URL, "clean/delete");
Review Comment:
Already update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058485340
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -277,7 +277,7 @@ protected void writeTableMetadata(HoodieTable table, String instantTime, String
* Initialized the metadata table on start up, should only be called once on driver.
*/
public void initMetadataTable() {
- ((HoodieFlinkTableServiceClient<T>) tableServiceClient).initMetadataTable();
+ ((HoodieFlinkHoodieTableServiceClient<T>) tableServiceClient).initMetadataTable();
Review Comment:
it has the same problem with Java and Spark table service client classes. It should be HoodieJavaTableServiceClient and HoodieSparkRDDTableServiceClient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366779990
Add BaseTableServiceClient.
Move some method from BaseHoodieWriteClient to BaseTableServiceClient.
- asyncClean
- asyncArchive
- inlineCompaction(org.apache.hudi.common.util.Option<java.util.Map<java.lang.String,java.lang.String>>)
- inlineCompaction(org.apache.hudi.table.HoodieTable, org.apache.hudi.common.util.Option<java.util.Map<java.lang.String,java.lang.String>>)
- logCompact(java.lang.String, boolean)
- inlineLogCompact
- runAnyPendingCompactions
- runAnyPendingLogCompactions
- inlineScheduleCompaction
- scheduleCompaction
- compact
- commitCompaction
- completeCompaction
- scheduleLogCompaction
- scheduleLogCompactionAtInstant
- logCompact(java.lang.String)
- completeLogCompaction
- scheduleCompactionAtInstant
- scheduleClustering
- scheduleClusteringAtInstant
- scheduleCleaning
- scheduleCleaningAtInstant
- cluster
- runTableServicesInline
- scheduleTableService
- scheduleTableServiceInternal
- inlineClustering(org.apache.hudi.common.util.Option<java.util.Map<java.lang.String,java.lang.String>>)
- inlineClustering(org.apache.hudi.table.HoodieTable, org.apache.hudi.common.util.Option<java.util.Map<java.lang.String,java.lang.String>>)
- inlineScheduleClustering
- runAnyPendingClustering
- finalizeWrite
- writeTableMetadata
- clean
- archive
- getInflightTimelineExcludeCompactionAndClustering
- getPendingRollbackInfo(org.apache.hudi.common.table.HoodieTableMetaClient, java.lang.String)
- getPendingRollbackInfo(org.apache.hudi.common.table.HoodieTableMetaClient, java.lang.String, boolean)
- getPendingRollbackInfos(org.apache.hudi.common.table.HoodieTableMetaClient)
- getPendingRollbackInfos(org.apache.hudi.common.table.HoodieTableMetaClient, boolean)
- rollbackFailedWrites()
- rollbackFailedWrites(boolean)
- rollbackFailedWrites(java.util.Map<java.lang.String,org.apache.hudi.common.util.Option<org.apache.hudi.common.HoodiePendingRollbackInfo>>, boolean)
- getInstantsToRollback
- rollback
- rollbackFailedBootstrap
Move HoodieMetrics and TransactionManager from BaseHoodieWriteClient to BaseHoodieClient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1364000527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
}, {
"hash" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939",
"triggerID" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"triggerType" : "PUSH"
}, {
"hash" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* a614791ea45854805a892bf53ec99f0dcdad30c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939)
* 24f7f28bb6f5d124e250afae9e5a93b51cb0bd07 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058027348
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_DEPLOY_PARALLELISM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.parallelism")
+ .defaultValue(100)
+ .withDocumentation("The max parallelism to deploy for table service of this table, default 100");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXECUTION_ENGINE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".execution.engine")
+ .defaultValue("spark")
+ .withDocumentation("The execution engine to deploy for table service of this table, default spark");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.extra.params")
+ .defaultValue("")
+ .withDocumentation("The extra params to deploy for table service of this table, split by ';'");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TIMEOUT = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".timeout")
+ .defaultValue(300)
+ .withDocumentation("Connection timeout for client");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRIES = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retries")
+ .defaultValue(3)
+ .withDocumentation("Number of retries while opening a connection to table service manager");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRY_DELAY = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retry.delay")
+ .defaultValue(1)
+ .withDocumentation("Number of seconds for the client to wait between consecutive connection attempts");
+
+ public static final ConfigProperty<String> RETRY_EXCEPTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".operation.retry.exceptions")
+ .defaultValue("IOException")
+ .withDocumentation("The class name of the Exception that needs to be re-tryed, separated by commas. "
+ + "Default is empty which means retry all the IOException and RuntimeException from FileSystem");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TOLERABLE_NUM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".tolerable.num")
+ .defaultValue(0)
+ .withDocumentation("Number of connection to table manager service unsuccessful tolerable for the client");
Review Comment:
Will cause the write job to fail, I will add it to the doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366053993
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994",
"triggerID" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 022edcdb913eff2db16c824095f5e836a81c3b44 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374727149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1383816899
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14234",
"triggerID" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"triggerType" : "PUSH"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4701cd7f6ba883bca09d1c99b32135bf827c58d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14234)
* 1f27b8bccfd4bfd64c142c94fe50660ac9644ff2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253496993
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78f3167ee6aba247ef55dcfacf8c2be603d621be Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1264932668
@yuzhaojing please also fill up the PR description properly. as discussed, a class diagram to show the new hierarchy expedites the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1277395144
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b35c4e70a6564f5524b1e662142b51d419ccaeff Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181)
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 175998dab136062e0244c857fb3ef8982ceeada7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1278417924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 175998dab136062e0244c857fb3ef8982ceeada7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185)
* 011023c6bb6d766a7b3098312ff1352ebe05136f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1305497459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 7cf07c0681d32e8831af9f39168695d5540af636 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1305495583
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1311429769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 36e29156d84985f88e46463b6177e3e751bbe013 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879)
* 27ae850689d655b0054d4a106c10b418ff662fda UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1311512210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 36e29156d84985f88e46463b6177e3e751bbe013 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879)
* 27ae850689d655b0054d4a106c10b418ff662fda Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1311667968
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 27ae850689d655b0054d4a106c10b418ff662fda Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r995431902
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseTableServiceClient.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+
+public abstract class BaseTableServiceClient<O> extends CommonHoodieClient {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected BaseTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig, HoodieMetrics metrics) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected boolean tableServicesEnabled(HoodieWriteConfig config) {
+ boolean enabled = config.areTableServicesEnabled();
+ if (!enabled) {
+ LOG.warn(String.format("Table services are disabled. Set `%s` to enable.", HoodieWriteConfig.TABLE_SERVICES_ENABLED));
+ }
+ return enabled;
+ }
+
+ protected boolean delegateToTableManagerService(HoodieWriteConfig config, ActionType actionType) {
Review Comment:
Already update.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseTableServiceClient.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+
+public abstract class BaseTableServiceClient<O> extends CommonHoodieClient {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected BaseTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig, HoodieMetrics metrics) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected boolean tableServicesEnabled(HoodieWriteConfig config) {
+ boolean enabled = config.areTableServicesEnabled();
+ if (!enabled) {
+ LOG.warn(String.format("Table services are disabled. Set `%s` to enable.", HoodieWriteConfig.TABLE_SERVICES_ENABLED));
+ }
+ return enabled;
+ }
+
+ protected boolean delegateToTableManagerService(HoodieWriteConfig config, ActionType actionType) {
+ boolean supportsAction = config.getTableManagerConfig().isTableManagerSupportsAction(actionType);
+ if (supportsAction) {
+ LOG.warn(actionType.name() + " delegate to table manager service!");
Review Comment:
Already update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1277313257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78f3167ee6aba247ef55dcfacf8c2be603d621be Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549)
* b35c4e70a6564f5524b1e662142b51d419ccaeff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181)
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r995439104
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -270,51 +254,8 @@ public void preWrite(String instantTime, WriteOperationType writeOperationType,
// remove the async cleaning
}
- @Override
- protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
Review Comment:
Already update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1277405890
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b35c4e70a6564f5524b1e662142b51d419ccaeff Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181)
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 175998dab136062e0244c857fb3ef8982ceeada7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r995432226
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/RunsTableService.java:
##########
@@ -34,4 +35,12 @@ default boolean tableServicesEnabled(HoodieWriteConfig config) {
}
return enabled;
}
+
+ default boolean delegateToTableManagerService(HoodieWriteConfig config, ActionType actionType) {
+ boolean supportsAction = config.getTableManagerConfig().isTableManagerSupportsAction(actionType);
+ if (supportsAction) {
+ LOG.warn(actionType.name() + " delegate to table manager service!");
+ }
+ return supportsAction;
Review Comment:
Already update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1297020491
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689)
* 7cf07c0681d32e8831af9f39168695d5540af636 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1306710784
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 0da5a658baa0616cfb0666fe6a1d0d916191d274 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855)
* f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1375039153
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1373397662
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144",
"triggerID" : "1373391935",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1373391935",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366050003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"triggerType" : "PUSH"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 022edcdb913eff2db16c824095f5e836a81c3b44 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058027348
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_DEPLOY_PARALLELISM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.parallelism")
+ .defaultValue(100)
+ .withDocumentation("The max parallelism to deploy for table service of this table, default 100");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXECUTION_ENGINE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".execution.engine")
+ .defaultValue("spark")
+ .withDocumentation("The execution engine to deploy for table service of this table, default spark");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.extra.params")
+ .defaultValue("")
+ .withDocumentation("The extra params to deploy for table service of this table, split by ';'");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TIMEOUT = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".timeout")
+ .defaultValue(300)
+ .withDocumentation("Connection timeout for client");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRIES = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retries")
+ .defaultValue(3)
+ .withDocumentation("Number of retries while opening a connection to table service manager");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRY_DELAY = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retry.delay")
+ .defaultValue(1)
+ .withDocumentation("Number of seconds for the client to wait between consecutive connection attempts");
+
+ public static final ConfigProperty<String> RETRY_EXCEPTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".operation.retry.exceptions")
+ .defaultValue("IOException")
+ .withDocumentation("The class name of the Exception that needs to be re-tryed, separated by commas. "
+ + "Default is empty which means retry all the IOException and RuntimeException from FileSystem");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TOLERABLE_NUM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".tolerable.num")
+ .defaultValue(0)
+ .withDocumentation("Number of connection to table manager service unsuccessful tolerable for the client");
Review Comment:
Will cause the write task to fail, I will add it to the doc.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_DEPLOY_PARALLELISM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.parallelism")
+ .defaultValue(100)
+ .withDocumentation("The max parallelism to deploy for table service of this table, default 100");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXECUTION_ENGINE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".execution.engine")
+ .defaultValue("spark")
+ .withDocumentation("The execution engine to deploy for table service of this table, default spark");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.extra.params")
+ .defaultValue("")
Review Comment:
will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058026622
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -188,6 +188,7 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setTableCreateSchema(conf.getString(FlinkOptions.SOURCE_AVRO_SCHEMA))
.setTableType(conf.getString(FlinkOptions.TABLE_TYPE))
.setTableName(conf.getString(FlinkOptions.TABLE_NAME))
+ .setDatabaseName(conf.getString(FlinkOptions.DATABASE_NAME))
Review Comment:
Yes, this parameter was not passed on the flink side before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1363688752
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
}, {
"hash" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* f7e140b4aff4e87ca18e8390faa04e1c72b92378 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916)
* a614791ea45854805a892bf53ec99f0dcdad30c6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058484066
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -277,7 +277,7 @@ protected void writeTableMetadata(HoodieTable table, String instantTime, String
* Initialized the metadata table on start up, should only be called once on driver.
*/
public void initMetadataTable() {
- ((HoodieFlinkTableServiceClient<T>) tableServiceClient).initMetadataTable();
+ ((HoodieFlinkHoodieTableServiceClient<T>) tableServiceClient).initMetadataTable();
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1062116644
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -0,0 +1,840 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class BaseHoodieTableServiceClient<O> extends BaseHoodieClient implements RunsTableService {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+
+ protected BaseHoodieTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected void startAsyncCleanerService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncCleanerService == null) {
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(writeClient);
+ } else {
+ this.asyncCleanerService.start(null);
+ }
+ }
+
+ protected void startAsyncArchiveService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncArchiveService == null) {
+ this.asyncArchiveService = AsyncArchiveService.startAsyncArchiveIfEnabled(writeClient);
+ } else {
+ this.asyncArchiveService.start(null);
+ }
+ }
+
+ protected void asyncClean() {
+ AsyncCleanerService.waitForCompletion(asyncCleanerService);
+ }
+
+ protected void asyncArchive() {
+ AsyncArchiveService.waitForCompletion(asyncArchiveService);
+ }
+
+ protected void setTableServiceTimer(WriteOperationType operationType) {
+ switch (operationType) {
+ case CLUSTER:
+ clusteringTimer = metrics.getClusteringCtx();
+ break;
+ case COMPACT:
+ compactionTimer = metrics.getCompactionCtx();
+ break;
+ case LOG_COMPACT:
+ logCompactionTimer = metrics.getLogCompactionCtx();
+ break;
+ default:
+ }
+ }
+
+ /**
+ * Performs a compaction operation on a table, serially before or after an insert/upsert action.
+ * Scheduling and execution is done inline.
+ */
+ protected Option<String> inlineCompaction(Option<Map<String, String>> extraMetadata) {
+ Option<String> compactionInstantTimeOpt = inlineScheduleCompaction(extraMetadata);
+ compactionInstantTimeOpt.ifPresent(compactInstantTime -> {
+ // inline compaction should auto commit as the user is never given control
+ compact(compactInstantTime, true);
+ });
+ return compactionInstantTimeOpt;
+ }
+
+ private void inlineCompaction(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.compaction)) {
+ scheduleCompaction(extraMetadata);
+ } else {
+ runAnyPendingCompactions(table);
+ inlineCompaction(extraMetadata);
+ }
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Log Compaction for the workload stored in instant-time.s
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected HoodieWriteMetadata<O> logCompact(String compactionInstantTime, boolean shouldComplete) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+ /**
+ * Performs a log compaction operation on a table, serially before or after an insert/upsert action.
+ */
+ protected Option<String> inlineLogCompact(Option<Map<String, String>> extraMetadata) {
+ Option<String> logCompactionInstantTimeOpt = scheduleLogCompaction(extraMetadata);
+ logCompactionInstantTimeOpt.ifPresent(logCompactInstantTime -> {
+ // inline log compaction should auto commit as the user is never given control
+ logCompact(logCompactInstantTime, true);
+ });
+ return logCompactionInstantTimeOpt;
+ }
+
+ protected void runAnyPendingCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight compaction at instant " + instant);
+ compact(instant.getTimestamp(), true);
+ });
+ }
+
+ protected void runAnyPendingLogCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingLogCompactionTimeline().getInstantsAsStream()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight log compaction at instant " + instant);
+ logCompact(instant.getTimestamp(), true);
+ });
+ }
+
+ /***
+ * Schedules compaction inline.
+ * @param extraMetadata extrametada to be used.
+ * @return compaction instant if scheduled.
+ */
+ protected Option<String> inlineScheduleCompaction(Option<Map<String, String>> extraMetadata) {
+ return scheduleCompaction(extraMetadata);
+ }
+
+ /**
+ * Schedules a new compaction instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected abstract HoodieWriteMetadata<O> compact(String compactionInstantTime, boolean shouldComplete);
+
+ /**
+ * Commit a compaction operation. Allow passing additional meta-data to be stored in commit instant file.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @param metadata All the metadata that gets stored along with a commit
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public abstract void commitCompaction(String compactionInstantTime, HoodieCommitMetadata metadata,
+ Option<Map<String, String>> extraMetadata);
+
+ /**
+ * Commit Compaction and track metrics.
+ */
+ protected abstract void completeCompaction(HoodieCommitMetadata metadata, HoodieTable table, String compactionCommitTime);
+
+ /**
+ * Schedules a new log compaction instant.
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleLogCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleLogCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new log compaction instant with passed-in instant time.
+ * @param instantTime Log Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleLogCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.LOG_COMPACT).isPresent();
+ }
+
+ /**
+ * Performs Log Compaction for the workload stored in instant-time.
+ *
+ * @param logCompactionInstantTime Log Compaction Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> logCompact(String logCompactionInstantTime) {
+ return logCompact(logCompactionInstantTime, config.shouldAutoCommit());
+ }
+
+ /**
+ * Commit Log Compaction and track metrics.
+ */
+ protected void completeLogCompaction(HoodieCommitMetadata metadata, HoodieTable table, String logCompactionCommitTime) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+
+ /**
+ * Schedules a new compaction instant with passed-in instant time.
+ *
+ * @param instantTime Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.COMPACT).isPresent();
+ }
+
+ /**
+ * Schedules a new clustering instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleClustering(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleClusteringAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new clustering instant with passed-in instant time.
+ *
+ * @param instantTime clustering Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleClusteringAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLUSTER).isPresent();
+ }
+
+ /**
+ * Schedules a new cleaning instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected Option<String> scheduleCleaning(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCleaningAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new cleaning instant with passed-in instant time.
+ *
+ * @param instantTime cleaning Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected boolean scheduleCleaningAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLEAN).isPresent();
+ }
+
+ /**
+ * Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.
+ *
+ * @param clusteringInstant Clustering Instant Time
+ * @return Collection of Write Status
+ */
+ public abstract HoodieWriteMetadata<O> cluster(String clusteringInstant, boolean shouldComplete);
+
+ protected void runTableServicesInline(HoodieTable table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
+ if (!tableServicesEnabled(config)) {
+ return;
+ }
+
+ if (!config.areAnyTableServicesExecutedInline() && !config.areAnyTableServicesScheduledInline()) {
+ return;
+ }
+
+ if (config.isMetadataTableEnabled()) {
+ table.getHoodieView().sync();
+ }
+ // Do an inline compaction if enabled
+ if (config.inlineCompactionEnabled()) {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
+ inlineCompaction(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().empty()) {
+ // proceed only if there are no pending compactions
+ metadata.addMetadata(HoodieCompactionConfig.SCHEDULE_INLINE_COMPACT.key(), "true");
+ inlineScheduleCompaction(extraMetadata);
+ }
+
+ // Do an inline log compaction if enabled
+ if (config.inlineLogCompactionEnabled()) {
+ runAnyPendingLogCompactions(table);
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "true");
+ inlineLogCompact(extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "false");
+ }
+
+ // Do an inline clustering if enabled
+ if (config.inlineClusteringEnabled()) {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "true");
+ inlineClustering(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineClusteringEnabled() && config.scheduleInlineClustering()
+ && table.getActiveTimeline().filterPendingReplaceTimeline().empty()) {
+ // proceed only if there are no pending clustering
+ metadata.addMetadata(HoodieClusteringConfig.SCHEDULE_INLINE_CLUSTERING.key(), "true");
+ inlineScheduleClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedule table services such as clustering, compaction & cleaning.
+ *
+ * @param extraMetadata Metadata to pass onto the scheduled service instant
+ * @param tableServiceType Type of table service to schedule
+ * @return
+ */
+ public Option<String> scheduleTableService(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ // A lock is required to guard against race conditions between an on-going writer and scheduling a table service.
+ final Option<HoodieInstant> inflightInstant = Option.of(new HoodieInstant(HoodieInstant.State.REQUESTED,
+ tableServiceType.getAction(), instantTime));
+ try {
+ this.txnManager.beginTransaction(inflightInstant, Option.empty());
+ LOG.info("Scheduling table service " + tableServiceType);
+ return scheduleTableServiceInternal(instantTime, extraMetadata, tableServiceType);
+ } finally {
+ this.txnManager.endTransaction(inflightInstant);
+ }
+ }
+
+ protected Option<String> scheduleTableServiceInternal(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ if (!tableServicesEnabled(config)) {
+ return Option.empty();
+ }
+
+ Option<String> option = Option.empty();
+ HoodieTable<?, ?, ?, ?> table = createTable(config, hadoopConf);
+
+ switch (tableServiceType) {
+ case ARCHIVE:
+ LOG.info("Scheduling archiving is not supported. Skipping.");
+ break;
+ case CLUSTER:
+ LOG.info("Scheduling clustering at instant time :" + instantTime);
+ Option<HoodieClusteringPlan> clusteringPlan = table
+ .scheduleClustering(context, instantTime, extraMetadata);
+ option = clusteringPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case COMPACT:
+ LOG.info("Scheduling compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> compactionPlan = table
+ .scheduleCompaction(context, instantTime, extraMetadata);
+ option = compactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case LOG_COMPACT:
+ LOG.info("Scheduling log compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> logCompactionPlan = table
+ .scheduleLogCompaction(context, instantTime, extraMetadata);
+ option = logCompactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case CLEAN:
+ LOG.info("Scheduling cleaning at instant time :" + instantTime);
+ Option<HoodieCleanerPlan> cleanerPlan = table
+ .scheduleCleaning(context, instantTime, extraMetadata);
+ option = cleanerPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ default:
+ throw new IllegalArgumentException("Invalid TableService " + tableServiceType);
+ }
+
+ Option<String> instantRange = delegateToTableServiceManager(tableServiceType, table);
+ if (instantRange.isPresent()) {
+ LOG.info("Delegate instant [" + instantRange.get() + "] to table service manager");
+ }
+
+ return option;
+ }
+
+ protected abstract HoodieTable<?, ?, ?, ?> createTable(HoodieWriteConfig config, Configuration hadoopConf);
+
+ /**
+ * Executes a clustering plan on a table, serially before or after an insert/upsert action.
+ * Schedules and executes clustering inline.
+ */
+ protected Option<String> inlineClustering(Option<Map<String, String>> extraMetadata) {
+ Option<String> clusteringInstantOpt = inlineScheduleClustering(extraMetadata);
+ clusteringInstantOpt.ifPresent(clusteringInstant -> {
+ // inline cluster should auto commit as the user is never given control
+ cluster(clusteringInstant, true);
+ });
+ return clusteringInstantOpt;
+ }
+
+ private void inlineClustering(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.replacecommit)) {
+ scheduleClustering(extraMetadata);
+ } else {
+ runAnyPendingClustering(table);
+ inlineClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedules clustering inline.
+ *
+ * @param extraMetadata extrametadata to use.
+ * @return clustering instant if scheduled.
+ */
+ protected Option<String> inlineScheduleClustering(Option<Map<String, String>> extraMetadata) {
+ return scheduleClustering(extraMetadata);
+ }
+
+ protected void runAnyPendingClustering(HoodieTable table) {
+ table.getActiveTimeline().filterPendingReplaceTimeline().getInstants().forEach(instant -> {
+ Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(table.getMetaClient(), instant);
+ if (instantPlan.isPresent()) {
+ LOG.info("Running pending clustering at instant " + instantPlan.get().getLeft());
+ cluster(instant.getTimestamp(), true);
+ }
+ });
+ }
+
+ /**
+ * Finalize Write operation.
+ *
+ * @param table HoodieTable
+ * @param instantTime Instant Time
+ * @param stats Hoodie Write Stat
+ */
+ protected void finalizeWrite(HoodieTable table, String instantTime, List<HoodieWriteStat> stats) {
Review Comment:
This is a good idea and I will extract it to BaseHoodieClient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1372197578
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114",
"triggerID" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"triggerType" : "PUSH"
}, {
"hash" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14121",
"triggerID" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dff9924f25bf457dc12cdd7a7a11b923ad08c3b7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14121)
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1384279123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 1f27b8bccfd4bfd64c142c94fe50660ac9644ff2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347)
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253214265
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 195a76d8e8369bce14d09ac2b46bfb293eb858ae Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544)
* 149e8f3c61e1221c4659da00a4168faeed1174dd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1277388505
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b35c4e70a6564f5524b1e662142b51d419ccaeff Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181)
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r995431711
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -125,7 +114,7 @@
* @param <K> Type of keys
* @param <O> Type of outputs
*/
-public abstract class BaseHoodieWriteClient<T extends HoodieRecordPayload, I, K, O> extends BaseHoodieClient
+public abstract class BaseHoodieWriteClient<T extends HoodieRecordPayload, I, K, O> extends CommonHoodieClient
Review Comment:
Already update.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseTableServiceClient.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+
+public abstract class BaseTableServiceClient<O> extends CommonHoodieClient {
Review Comment:
Already update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1278650330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 011023c6bb6d766a7b3098312ff1352ebe05136f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196)
* d4408906bcc13c55f6350f35c168be32d978daf6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1296863896
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* d4408906bcc13c55f6350f35c168be32d978daf6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204)
* df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366140007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994",
"triggerID" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 022edcdb913eff2db16c824095f5e836a81c3b44 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058454424
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
Review Comment:
we also support java engine. so do you agree with this pattern with engine prefix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1364306798
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
}, {
"hash" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939",
"triggerID" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"triggerType" : "PUSH"
}, {
"hash" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13943",
"triggerID" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* 097828528e3529f0590d38be30e27e2787a64fa2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1362982879
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 27ae850689d655b0054d4a106c10b418ff662fda Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949)
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058027049
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
Review Comment:
Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058026788
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
Review Comment:
will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1378226494
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14234",
"triggerID" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190)
* e4701cd7f6ba883bca09d1c99b32135bf827c58d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14234)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1373196708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135)
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1373960675
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144",
"triggerID" : "1373391935",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1373391935",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1062117175
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -0,0 +1,840 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class BaseHoodieTableServiceClient<O> extends BaseHoodieClient implements RunsTableService {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+
+ protected BaseHoodieTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected void startAsyncCleanerService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncCleanerService == null) {
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(writeClient);
+ } else {
+ this.asyncCleanerService.start(null);
+ }
+ }
+
+ protected void startAsyncArchiveService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncArchiveService == null) {
+ this.asyncArchiveService = AsyncArchiveService.startAsyncArchiveIfEnabled(writeClient);
+ } else {
+ this.asyncArchiveService.start(null);
+ }
+ }
+
+ protected void asyncClean() {
+ AsyncCleanerService.waitForCompletion(asyncCleanerService);
+ }
+
+ protected void asyncArchive() {
+ AsyncArchiveService.waitForCompletion(asyncArchiveService);
+ }
+
+ protected void setTableServiceTimer(WriteOperationType operationType) {
+ switch (operationType) {
+ case CLUSTER:
+ clusteringTimer = metrics.getClusteringCtx();
+ break;
+ case COMPACT:
+ compactionTimer = metrics.getCompactionCtx();
+ break;
+ case LOG_COMPACT:
+ logCompactionTimer = metrics.getLogCompactionCtx();
+ break;
+ default:
+ }
+ }
+
+ /**
+ * Performs a compaction operation on a table, serially before or after an insert/upsert action.
+ * Scheduling and execution is done inline.
+ */
+ protected Option<String> inlineCompaction(Option<Map<String, String>> extraMetadata) {
+ Option<String> compactionInstantTimeOpt = inlineScheduleCompaction(extraMetadata);
+ compactionInstantTimeOpt.ifPresent(compactInstantTime -> {
+ // inline compaction should auto commit as the user is never given control
+ compact(compactInstantTime, true);
+ });
+ return compactionInstantTimeOpt;
+ }
+
+ private void inlineCompaction(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.compaction)) {
+ scheduleCompaction(extraMetadata);
+ } else {
+ runAnyPendingCompactions(table);
+ inlineCompaction(extraMetadata);
+ }
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Log Compaction for the workload stored in instant-time.s
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected HoodieWriteMetadata<O> logCompact(String compactionInstantTime, boolean shouldComplete) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+ /**
+ * Performs a log compaction operation on a table, serially before or after an insert/upsert action.
+ */
+ protected Option<String> inlineLogCompact(Option<Map<String, String>> extraMetadata) {
+ Option<String> logCompactionInstantTimeOpt = scheduleLogCompaction(extraMetadata);
+ logCompactionInstantTimeOpt.ifPresent(logCompactInstantTime -> {
+ // inline log compaction should auto commit as the user is never given control
+ logCompact(logCompactInstantTime, true);
+ });
+ return logCompactionInstantTimeOpt;
+ }
+
+ protected void runAnyPendingCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight compaction at instant " + instant);
+ compact(instant.getTimestamp(), true);
+ });
+ }
+
+ protected void runAnyPendingLogCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingLogCompactionTimeline().getInstantsAsStream()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight log compaction at instant " + instant);
+ logCompact(instant.getTimestamp(), true);
+ });
+ }
+
+ /***
+ * Schedules compaction inline.
+ * @param extraMetadata extrametada to be used.
+ * @return compaction instant if scheduled.
+ */
+ protected Option<String> inlineScheduleCompaction(Option<Map<String, String>> extraMetadata) {
+ return scheduleCompaction(extraMetadata);
+ }
+
+ /**
+ * Schedules a new compaction instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected abstract HoodieWriteMetadata<O> compact(String compactionInstantTime, boolean shouldComplete);
+
+ /**
+ * Commit a compaction operation. Allow passing additional meta-data to be stored in commit instant file.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @param metadata All the metadata that gets stored along with a commit
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public abstract void commitCompaction(String compactionInstantTime, HoodieCommitMetadata metadata,
+ Option<Map<String, String>> extraMetadata);
+
+ /**
+ * Commit Compaction and track metrics.
+ */
+ protected abstract void completeCompaction(HoodieCommitMetadata metadata, HoodieTable table, String compactionCommitTime);
+
+ /**
+ * Schedules a new log compaction instant.
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleLogCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleLogCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new log compaction instant with passed-in instant time.
+ * @param instantTime Log Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleLogCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.LOG_COMPACT).isPresent();
+ }
+
+ /**
+ * Performs Log Compaction for the workload stored in instant-time.
+ *
+ * @param logCompactionInstantTime Log Compaction Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> logCompact(String logCompactionInstantTime) {
+ return logCompact(logCompactionInstantTime, config.shouldAutoCommit());
+ }
+
+ /**
+ * Commit Log Compaction and track metrics.
+ */
+ protected void completeLogCompaction(HoodieCommitMetadata metadata, HoodieTable table, String logCompactionCommitTime) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+
+ /**
+ * Schedules a new compaction instant with passed-in instant time.
+ *
+ * @param instantTime Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.COMPACT).isPresent();
+ }
+
+ /**
+ * Schedules a new clustering instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleClustering(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleClusteringAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new clustering instant with passed-in instant time.
+ *
+ * @param instantTime clustering Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleClusteringAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLUSTER).isPresent();
+ }
+
+ /**
+ * Schedules a new cleaning instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected Option<String> scheduleCleaning(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCleaningAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new cleaning instant with passed-in instant time.
+ *
+ * @param instantTime cleaning Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected boolean scheduleCleaningAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLEAN).isPresent();
+ }
+
+ /**
+ * Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.
+ *
+ * @param clusteringInstant Clustering Instant Time
+ * @return Collection of Write Status
+ */
+ public abstract HoodieWriteMetadata<O> cluster(String clusteringInstant, boolean shouldComplete);
+
+ protected void runTableServicesInline(HoodieTable table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
+ if (!tableServicesEnabled(config)) {
+ return;
+ }
+
+ if (!config.areAnyTableServicesExecutedInline() && !config.areAnyTableServicesScheduledInline()) {
+ return;
+ }
+
+ if (config.isMetadataTableEnabled()) {
+ table.getHoodieView().sync();
+ }
+ // Do an inline compaction if enabled
+ if (config.inlineCompactionEnabled()) {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
+ inlineCompaction(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().empty()) {
+ // proceed only if there are no pending compactions
+ metadata.addMetadata(HoodieCompactionConfig.SCHEDULE_INLINE_COMPACT.key(), "true");
+ inlineScheduleCompaction(extraMetadata);
+ }
+
+ // Do an inline log compaction if enabled
+ if (config.inlineLogCompactionEnabled()) {
+ runAnyPendingLogCompactions(table);
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "true");
+ inlineLogCompact(extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "false");
+ }
+
+ // Do an inline clustering if enabled
+ if (config.inlineClusteringEnabled()) {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "true");
+ inlineClustering(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineClusteringEnabled() && config.scheduleInlineClustering()
+ && table.getActiveTimeline().filterPendingReplaceTimeline().empty()) {
+ // proceed only if there are no pending clustering
+ metadata.addMetadata(HoodieClusteringConfig.SCHEDULE_INLINE_CLUSTERING.key(), "true");
+ inlineScheduleClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedule table services such as clustering, compaction & cleaning.
+ *
+ * @param extraMetadata Metadata to pass onto the scheduled service instant
+ * @param tableServiceType Type of table service to schedule
+ * @return
+ */
+ public Option<String> scheduleTableService(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ // A lock is required to guard against race conditions between an on-going writer and scheduling a table service.
+ final Option<HoodieInstant> inflightInstant = Option.of(new HoodieInstant(HoodieInstant.State.REQUESTED,
+ tableServiceType.getAction(), instantTime));
+ try {
+ this.txnManager.beginTransaction(inflightInstant, Option.empty());
+ LOG.info("Scheduling table service " + tableServiceType);
+ return scheduleTableServiceInternal(instantTime, extraMetadata, tableServiceType);
+ } finally {
+ this.txnManager.endTransaction(inflightInstant);
+ }
+ }
+
+ protected Option<String> scheduleTableServiceInternal(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ if (!tableServicesEnabled(config)) {
+ return Option.empty();
+ }
+
+ Option<String> option = Option.empty();
+ HoodieTable<?, ?, ?, ?> table = createTable(config, hadoopConf);
+
+ switch (tableServiceType) {
+ case ARCHIVE:
+ LOG.info("Scheduling archiving is not supported. Skipping.");
+ break;
+ case CLUSTER:
+ LOG.info("Scheduling clustering at instant time :" + instantTime);
+ Option<HoodieClusteringPlan> clusteringPlan = table
+ .scheduleClustering(context, instantTime, extraMetadata);
+ option = clusteringPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case COMPACT:
+ LOG.info("Scheduling compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> compactionPlan = table
+ .scheduleCompaction(context, instantTime, extraMetadata);
+ option = compactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case LOG_COMPACT:
+ LOG.info("Scheduling log compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> logCompactionPlan = table
+ .scheduleLogCompaction(context, instantTime, extraMetadata);
+ option = logCompactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case CLEAN:
+ LOG.info("Scheduling cleaning at instant time :" + instantTime);
+ Option<HoodieCleanerPlan> cleanerPlan = table
+ .scheduleCleaning(context, instantTime, extraMetadata);
+ option = cleanerPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ default:
+ throw new IllegalArgumentException("Invalid TableService " + tableServiceType);
+ }
+
+ Option<String> instantRange = delegateToTableServiceManager(tableServiceType, table);
+ if (instantRange.isPresent()) {
+ LOG.info("Delegate instant [" + instantRange.get() + "] to table service manager");
+ }
+
+ return option;
+ }
+
+ protected abstract HoodieTable<?, ?, ?, ?> createTable(HoodieWriteConfig config, Configuration hadoopConf);
+
+ /**
+ * Executes a clustering plan on a table, serially before or after an insert/upsert action.
+ * Schedules and executes clustering inline.
+ */
+ protected Option<String> inlineClustering(Option<Map<String, String>> extraMetadata) {
+ Option<String> clusteringInstantOpt = inlineScheduleClustering(extraMetadata);
+ clusteringInstantOpt.ifPresent(clusteringInstant -> {
+ // inline cluster should auto commit as the user is never given control
+ cluster(clusteringInstant, true);
+ });
+ return clusteringInstantOpt;
+ }
+
+ private void inlineClustering(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.replacecommit)) {
+ scheduleClustering(extraMetadata);
+ } else {
+ runAnyPendingClustering(table);
+ inlineClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedules clustering inline.
+ *
+ * @param extraMetadata extrametadata to use.
+ * @return clustering instant if scheduled.
+ */
+ protected Option<String> inlineScheduleClustering(Option<Map<String, String>> extraMetadata) {
+ return scheduleClustering(extraMetadata);
+ }
+
+ protected void runAnyPendingClustering(HoodieTable table) {
+ table.getActiveTimeline().filterPendingReplaceTimeline().getInstants().forEach(instant -> {
+ Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(table.getMetaClient(), instant);
+ if (instantPlan.isPresent()) {
+ LOG.info("Running pending clustering at instant " + instantPlan.get().getLeft());
+ cluster(instant.getTimestamp(), true);
+ }
+ });
+ }
+
+ /**
+ * Finalize Write operation.
+ *
+ * @param table HoodieTable
+ * @param instantTime Instant Time
+ * @param stats Hoodie Write Stat
+ */
+ protected void finalizeWrite(HoodieTable table, String instantTime, List<HoodieWriteStat> stats) {
+ try {
+ final Timer.Context finalizeCtx = metrics.getFinalizeCtx();
+ table.finalizeWrite(context, instantTime, stats);
+ if (finalizeCtx != null) {
+ Option<Long> durationInMs = Option.of(metrics.getDurationInMs(finalizeCtx.stop()));
+ durationInMs.ifPresent(duration -> {
+ LOG.info("Finalize write elapsed time (milliseconds): " + duration);
+ metrics.updateFinalizeWriteMetrics(duration, stats.size());
+ });
+ }
+ } catch (HoodieIOException ioe) {
+ throw new HoodieCommitException("Failed to complete commit " + instantTime + " due to finalize errors.", ioe);
+ }
+ }
+
+ /**
+ * Write the HoodieCommitMetadata to metadata table if available.
+ *
+ * @param table {@link HoodieTable} of interest.
+ * @param instantTime instant time of the commit.
+ * @param actionType action type of the commit.
+ * @param metadata instance of {@link HoodieCommitMetadata}.
+ */
+ protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
+ context.setJobStatus(this.getClass().getSimpleName(), "Committing to metadata table: " + config.getTableName());
+ table.getMetadataWriter(instantTime).ifPresent(w -> ((HoodieTableMetadataWriter) w).update(metadata, instantTime,
+ table.isTableServiceAction(actionType, instantTime)));
Review Comment:
This makes it clearer and easier to be compatible when subsequent table service operations may be inconsistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374118065
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1277581984
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 175998dab136062e0244c857fb3ef8982ceeada7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1297445204
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 7cf07c0681d32e8831af9f39168695d5540af636 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing closed pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing closed pull request #6732: [HUDI-4148] Add client for hudi table management service
URL: https://github.com/apache/hudi/pull/6732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1254681134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 78f3167ee6aba247ef55dcfacf8c2be603d621be Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253266426
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 195a76d8e8369bce14d09ac2b46bfb293eb858ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544)
* 149e8f3c61e1221c4659da00a4168faeed1174dd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547)
* 78f3167ee6aba247ef55dcfacf8c2be603d621be Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1254627475
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 78f3167ee6aba247ef55dcfacf8c2be603d621be Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058461044
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
Review Comment:
Agree it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366755308
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64ecea100e226b7fd539cab05c03bc9902e36db1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366226561
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994",
"triggerID" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd837f2febbd7c102305f91f8e85532782d29467 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1062133114
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -352,182 +303,23 @@ public void commitLogCompaction(String logCompactionInstantTime, HoodieCommitMet
protected void completeLogCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String logCompactionCommitTime) {
- this.context.setJobStatus(this.getClass().getSimpleName(), "Collect log compaction write status and commit compaction");
- List<HoodieWriteStat> writeStats = metadata.getWriteStats();
final HoodieInstant logCompactionInstant = new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.LOG_COMPACTION_ACTION, logCompactionCommitTime);
- try {
- this.txnManager.beginTransaction(Option.of(logCompactionInstant), Option.empty());
- preCommit(logCompactionInstant, metadata);
- finalizeWrite(table, logCompactionCommitTime, writeStats);
- // commit to data table after committing to metadata table.
- updateTableMetadata(table, metadata, logCompactionInstant);
- LOG.info("Committing Log Compaction " + logCompactionCommitTime + ". Finished with result " + metadata);
- CompactHelpers.getInstance().completeInflightLogCompaction(table, logCompactionCommitTime, metadata);
- } finally {
- this.txnManager.endTransaction(Option.of(logCompactionInstant));
- }
- WriteMarkersFactory.get(config.getMarkersType(), table, logCompactionCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
- if (compactionTimer != null) {
- long durationInMs = metrics.getDurationInMs(compactionTimer.stop());
- HoodieActiveTimeline.parseDateFromInstantTimeSafely(logCompactionCommitTime).ifPresent(parsedInstant ->
- metrics.updateCommitMetrics(parsedInstant.getTime(), durationInMs, metadata, HoodieActiveTimeline.LOG_COMPACTION_ACTION)
- );
- }
- LOG.info("Log Compacted successfully on commit " + logCompactionCommitTime);
+ preCommit(logCompactionInstant, metadata);
+ tableServiceClient.completeLogCompaction(metadata, table, logCompactionCommitTime);
}
@Override
protected HoodieWriteMetadata<JavaRDD<WriteStatus>> logCompact(String logCompactionInstantTime, boolean shouldComplete) {
HoodieSparkTable<T> table = HoodieSparkTable.create(config, context);
preWrite(logCompactionInstantTime, WriteOperationType.LOG_COMPACT, table.getMetaClient());
- HoodieTimeline pendingLogCompactionTimeline = table.getActiveTimeline().filterPendingLogCompactionTimeline();
- HoodieInstant inflightInstant = HoodieTimeline.getLogCompactionInflightInstant(logCompactionInstantTime);
- if (pendingLogCompactionTimeline.containsInstant(inflightInstant)) {
- LOG.info("Found Log compaction inflight file. Rolling back the commit and exiting.");
- table.rollbackInflightLogCompaction(inflightInstant, commitToRollback -> getPendingRollbackInfo(table.getMetaClient(), commitToRollback, false));
- table.getMetaClient().reloadActiveTimeline();
- throw new HoodieException("Inflight logcompaction file exists");
- }
- logCompactionTimer = metrics.getLogCompactionCtx();
- WriteMarkersFactory.get(config.getMarkersType(), table, logCompactionInstantTime);
- HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata = table.logCompact(context, logCompactionInstantTime);
- HoodieWriteMetadata<JavaRDD<WriteStatus>> logCompactionMetadata = writeMetadata.clone(HoodieJavaRDD.getJavaRDD(writeMetadata.getWriteStatuses()));
- if (shouldComplete && logCompactionMetadata.getCommitMetadata().isPresent()) {
- completeTableService(TableServiceType.LOG_COMPACT, logCompactionMetadata.getCommitMetadata().get(), table, logCompactionInstantTime);
- }
- return logCompactionMetadata;
+ return tableServiceClient.logCompact(logCompactionInstantTime, shouldComplete);
}
@Override
public HoodieWriteMetadata<JavaRDD<WriteStatus>> cluster(String clusteringInstant, boolean shouldComplete) {
HoodieSparkTable<T> table = HoodieSparkTable.create(config, context);
preWrite(clusteringInstant, WriteOperationType.CLUSTER, table.getMetaClient());
- HoodieTimeline pendingClusteringTimeline = table.getActiveTimeline().filterPendingReplaceTimeline();
- HoodieInstant inflightInstant = HoodieTimeline.getReplaceCommitInflightInstant(clusteringInstant);
- if (pendingClusteringTimeline.containsInstant(inflightInstant)) {
- table.rollbackInflightClustering(inflightInstant, commitToRollback -> getPendingRollbackInfo(table.getMetaClient(), commitToRollback, false));
- table.getMetaClient().reloadActiveTimeline();
- }
- clusteringTimer = metrics.getClusteringCtx();
- LOG.info("Starting clustering at " + clusteringInstant);
- HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata = table.cluster(context, clusteringInstant);
- HoodieWriteMetadata<JavaRDD<WriteStatus>> clusteringMetadata = writeMetadata.clone(HoodieJavaRDD.getJavaRDD(writeMetadata.getWriteStatuses()));
- // Validation has to be done after cloning. if not, it could result in dereferencing the write status twice which means clustering could get executed twice.
- validateClusteringCommit(clusteringMetadata, clusteringInstant, table);
- // TODO : Where is shouldComplete used ?
- if (shouldComplete && clusteringMetadata.getCommitMetadata().isPresent()) {
- completeTableService(TableServiceType.CLUSTER, clusteringMetadata.getCommitMetadata().get(), table, clusteringInstant);
- }
- return clusteringMetadata;
- }
-
- private void completeClustering(HoodieReplaceCommitMetadata metadata,
- HoodieTable table,
- String clusteringCommitTime) {
- List<HoodieWriteStat> writeStats = metadata.getPartitionToWriteStats().entrySet().stream().flatMap(e ->
- e.getValue().stream()).collect(Collectors.toList());
-
- if (writeStats.stream().mapToLong(HoodieWriteStat::getTotalWriteErrors).sum() > 0) {
- throw new HoodieClusteringException("Clustering failed to write to files:"
- + writeStats.stream().filter(s -> s.getTotalWriteErrors() > 0L).map(HoodieWriteStat::getFileId).collect(Collectors.joining(",")));
- }
-
- final HoodieInstant clusteringInstant = HoodieTimeline.getReplaceCommitInflightInstant(clusteringCommitTime);
- try {
- this.txnManager.beginTransaction(Option.of(clusteringInstant), Option.empty());
-
- finalizeWrite(table, clusteringCommitTime, writeStats);
- // Update table's metadata (table)
- updateTableMetadata(table, metadata, clusteringInstant);
-
- LOG.info("Committing Clustering " + clusteringCommitTime + ". Finished with result " + metadata);
-
- table.getActiveTimeline().transitionReplaceInflightToComplete(
- clusteringInstant,
- Option.of(metadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
- } catch (Exception e) {
- throw new HoodieClusteringException("unable to transition clustering inflight to complete: " + clusteringCommitTime, e);
- } finally {
- this.txnManager.endTransaction(Option.of(clusteringInstant));
- }
- WriteMarkersFactory.get(config.getMarkersType(), table, clusteringCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
- if (clusteringTimer != null) {
- long durationInMs = metrics.getDurationInMs(clusteringTimer.stop());
- HoodieActiveTimeline.parseDateFromInstantTimeSafely(clusteringCommitTime).ifPresent(parsedInstant ->
- metrics.updateCommitMetrics(parsedInstant.getTime(), durationInMs, metadata, HoodieActiveTimeline.REPLACE_COMMIT_ACTION)
- );
- }
- LOG.info("Clustering successfully on commit " + clusteringCommitTime);
- }
-
- private void validateClusteringCommit(HoodieWriteMetadata<JavaRDD<WriteStatus>> clusteringMetadata, String clusteringCommitTime, HoodieTable table) {
- if (clusteringMetadata.getWriteStatuses().isEmpty()) {
- HoodieClusteringPlan clusteringPlan = ClusteringUtils.getClusteringPlan(
- table.getMetaClient(), HoodieTimeline.getReplaceCommitRequestedInstant(clusteringCommitTime))
- .map(Pair::getRight).orElseThrow(() -> new HoodieClusteringException(
- "Unable to read clustering plan for instant: " + clusteringCommitTime));
- throw new HoodieClusteringException("Clustering plan produced 0 WriteStatus for " + clusteringCommitTime
- + " #groups: " + clusteringPlan.getInputGroups().size() + " expected at least "
- + clusteringPlan.getInputGroups().stream().mapToInt(HoodieClusteringGroup::getNumOutputFileGroups).sum()
- + " write statuses");
- }
- }
-
- private void updateTableMetadata(HoodieTable table, HoodieCommitMetadata commitMetadata,
- HoodieInstant hoodieInstant) {
- boolean isTableServiceAction = table.isTableServiceAction(hoodieInstant.getAction(), hoodieInstant.getTimestamp());
- // Do not do any conflict resolution here as we do with regular writes. We take the lock here to ensure all writes to metadata table happens within a
- // single lock (single writer). Because more than one write to metadata table will result in conflicts since all of them updates the same partition.
- table.getMetadataWriter(hoodieInstant.getTimestamp())
- .ifPresent(writer -> ((HoodieTableMetadataWriter) writer).update(commitMetadata, hoodieInstant.getTimestamp(), isTableServiceAction));
- }
-
- @Override
- protected void initMetadataTable(Option<String> instantTime) {
- // Initialize Metadata Table to make sure it's bootstrapped _before_ the operation,
- // if it didn't exist before
- // See https://issues.apache.org/jira/browse/HUDI-3343 for more details
- initializeMetadataTable(instantTime);
- }
-
- /**
- * Initialize the metadata table if needed. Creating the metadata table writer
- * will trigger the initial bootstrapping from the data table.
- *
- * @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
- */
- private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
- }
- }
- }
Review Comment:
This is a problem and I'm going to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1384622387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "6a422226c3cb64a13f41acf199c3b9b4ddbc7a30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14348",
"triggerID" : "6a422226c3cb64a13f41acf199c3b9b4ddbc7a30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a422226c3cb64a13f41acf199c3b9b4ddbc7a30 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14348)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1373199852
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374725718
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172)
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1384290337
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1383827979
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14234",
"triggerID" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"triggerType" : "PUSH"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4701cd7f6ba883bca09d1c99b32135bf827c58d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14234)
* 1f27b8bccfd4bfd64c142c94fe50660ac9644ff2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1384371065
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1384464837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "6a422226c3cb64a13f41acf199c3b9b4ddbc7a30",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14348",
"triggerID" : "6a422226c3cb64a13f41acf199c3b9b4ddbc7a30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a422226c3cb64a13f41acf199c3b9b4ddbc7a30 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14348)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253262365
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 195a76d8e8369bce14d09ac2b46bfb293eb858ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544)
* 149e8f3c61e1221c4659da00a4168faeed1174dd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547)
* 78f3167ee6aba247ef55dcfacf8c2be603d621be UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1254627373
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1278413158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 175998dab136062e0244c857fb3ef8982ceeada7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185)
* 011023c6bb6d766a7b3098312ff1352ebe05136f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1278607278
> @yuzhaojing please also fill up the PR description properly. as discussed, a class diagram to show the new hierarchy expedites the review.
Thanks for the reminder, the corresponding content has been added to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1305710308
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 7cf07c0681d32e8831af9f39168695d5540af636 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850)
* 0da5a658baa0616cfb0666fe6a1d0d916191d274 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1307201323
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869)
* 36e29156d84985f88e46463b6177e3e751bbe013 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1364940657
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1364935793
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 097828528e3529f0590d38be30e27e2787a64fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1062128020
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -352,182 +303,23 @@ public void commitLogCompaction(String logCompactionInstantTime, HoodieCommitMet
protected void completeLogCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String logCompactionCommitTime) {
- this.context.setJobStatus(this.getClass().getSimpleName(), "Collect log compaction write status and commit compaction");
- List<HoodieWriteStat> writeStats = metadata.getWriteStats();
final HoodieInstant logCompactionInstant = new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.LOG_COMPACTION_ACTION, logCompactionCommitTime);
- try {
- this.txnManager.beginTransaction(Option.of(logCompactionInstant), Option.empty());
- preCommit(logCompactionInstant, metadata);
- finalizeWrite(table, logCompactionCommitTime, writeStats);
- // commit to data table after committing to metadata table.
- updateTableMetadata(table, metadata, logCompactionInstant);
- LOG.info("Committing Log Compaction " + logCompactionCommitTime + ". Finished with result " + metadata);
- CompactHelpers.getInstance().completeInflightLogCompaction(table, logCompactionCommitTime, metadata);
- } finally {
- this.txnManager.endTransaction(Option.of(logCompactionInstant));
- }
- WriteMarkersFactory.get(config.getMarkersType(), table, logCompactionCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
- if (compactionTimer != null) {
- long durationInMs = metrics.getDurationInMs(compactionTimer.stop());
- HoodieActiveTimeline.parseDateFromInstantTimeSafely(logCompactionCommitTime).ifPresent(parsedInstant ->
- metrics.updateCommitMetrics(parsedInstant.getTime(), durationInMs, metadata, HoodieActiveTimeline.LOG_COMPACTION_ACTION)
- );
- }
- LOG.info("Log Compacted successfully on commit " + logCompactionCommitTime);
+ preCommit(logCompactionInstant, metadata);
+ tableServiceClient.completeLogCompaction(metadata, table, logCompactionCommitTime);
Review Comment:
This causes an instant unexpected problem with concurrency, which I need to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1372116504
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114",
"triggerID" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"triggerType" : "PUSH"
}, {
"hash" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28aacdc92d0f7027e03cb2cd8891b19b48e47da0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114)
* dff9924f25bf457dc12cdd7a7a11b923ad08c3b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1372182507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114",
"triggerID" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"triggerType" : "PUSH"
}, {
"hash" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14121",
"triggerID" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 28aacdc92d0f7027e03cb2cd8891b19b48e47da0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114)
* dff9924f25bf457dc12cdd7a7a11b923ad08c3b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14121)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1384459130
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14339",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14347",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "6a422226c3cb64a13f41acf199c3b9b4ddbc7a30",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6a422226c3cb64a13f41acf199c3b9b4ddbc7a30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6a422226c3cb64a13f41acf199c3b9b4ddbc7a30 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1385587792
the CI timeout issue happens on master and is unrelated to this PR itself. will land this first. CI issue will be addressed separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1057816433
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -188,6 +188,7 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setTableCreateSchema(conf.getString(FlinkOptions.SOURCE_AVRO_SCHEMA))
.setTableType(conf.getString(FlinkOptions.TABLE_TYPE))
.setTableName(conf.getString(FlinkOptions.TABLE_NAME))
+ .setDatabaseName(conf.getString(FlinkOptions.DATABASE_NAME))
Review Comment:
is this fixing a separate bug?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
Review Comment:
should make it `noDefaultValue()` if nothing should be the default
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -84,6 +84,12 @@ private FlinkOptions() {
// Common Options
// ------------------------------------------------------------------------
+ public static final ConfigOption<String> DATABASE_NAME = ConfigOptions
+ .key(HoodieTableConfig.DATABASE_NAME.key())
+ .stringType()
+ .noDefaultValue()
+ .withDescription("Database name to register to Hive metastore");
Review Comment:
can we separate this from refactoring changes?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
Review Comment:
this is specific to spark. so this string of resources representation is only parseable under the context of the engine? how about making it `spark:4g,4g` so we can merge the engine config into this
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_DEPLOY_PARALLELISM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.parallelism")
+ .defaultValue(100)
+ .withDocumentation("The max parallelism to deploy for table service of this table, default 100");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXECUTION_ENGINE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".execution.engine")
+ .defaultValue("spark")
+ .withDocumentation("The execution engine to deploy for table service of this table, default spark");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.extra.params")
+ .defaultValue("")
+ .withDocumentation("The extra params to deploy for table service of this table, split by ';'");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TIMEOUT = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".timeout")
+ .defaultValue(300)
+ .withDocumentation("Connection timeout for client");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRIES = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retries")
+ .defaultValue(3)
+ .withDocumentation("Number of retries while opening a connection to table service manager");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRY_DELAY = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retry.delay")
+ .defaultValue(1)
+ .withDocumentation("Number of seconds for the client to wait between consecutive connection attempts");
+
+ public static final ConfigProperty<String> RETRY_EXCEPTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".operation.retry.exceptions")
+ .defaultValue("IOException")
+ .withDocumentation("The class name of the Exception that needs to be re-tryed, separated by commas. "
+ + "Default is empty which means retry all the IOException and RuntimeException from FileSystem");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TOLERABLE_NUM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".tolerable.num")
+ .defaultValue(0)
+ .withDocumentation("Number of connection to table manager service unsuccessful tolerable for the client");
Review Comment:
this needs renaming. but first, can you clarify what will happen when number of unsuccessful connections reaches the threshold?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTableServiceManagerClient.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.common.config.HoodieTableServiceManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.RetryHelper;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Client which send the table service instants to the table service manager.
+ */
+public class HoodieTableServiceManagerClient {
+
+ /**
+ * Rollback commands, that trigger a specific handling for rollback.
+ */
Review Comment:
don't quite understand what this javadoc mean?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_DEPLOY_PARALLELISM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.parallelism")
+ .defaultValue(100)
+ .withDocumentation("The max parallelism to deploy for table service of this table, default 100");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXECUTION_ENGINE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".execution.engine")
+ .defaultValue("spark")
+ .withDocumentation("The execution engine to deploy for table service of this table, default spark");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.extra.params")
+ .defaultValue("")
+ .withDocumentation("The extra params to deploy for table service of this table, split by ';'");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_TIMEOUT = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".timeout")
+ .defaultValue(300)
+ .withDocumentation("Connection timeout for client");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRIES = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retries")
+ .defaultValue(3)
+ .withDocumentation("Number of retries while opening a connection to table service manager");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_RETRY_DELAY = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".connect.retry.delay")
+ .defaultValue(1)
+ .withDocumentation("Number of seconds for the client to wait between consecutive connection attempts");
+
+ public static final ConfigProperty<String> RETRY_EXCEPTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".operation.retry.exceptions")
+ .defaultValue("IOException")
Review Comment:
shouldn't the class name be fully qualified?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
+
+ public static final ConfigProperty<Integer> TABLE_SERVICE_MANAGER_DEPLOY_PARALLELISM = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.parallelism")
+ .defaultValue(100)
+ .withDocumentation("The max parallelism to deploy for table service of this table, default 100");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXECUTION_ENGINE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".execution.engine")
+ .defaultValue("spark")
+ .withDocumentation("The execution engine to deploy for table service of this table, default spark");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_EXTRA_PARAMS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.extra.params")
+ .defaultValue("")
Review Comment:
can we make it `noDefaultValue()` if nothing should be default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1367275891
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* c20aa589730546c0c7bb82969c92aa6d364af101 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253216943
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 195a76d8e8369bce14d09ac2b46bfb293eb858ae Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544)
* 149e8f3c61e1221c4659da00a4168faeed1174dd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253258327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 195a76d8e8369bce14d09ac2b46bfb293eb858ae Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544)
* 149e8f3c61e1221c4659da00a4168faeed1174dd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1062647677
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -188,7 +188,6 @@ public static HoodieTableMetaClient initTableIfNotExists(
.setTableCreateSchema(conf.getString(FlinkOptions.SOURCE_AVRO_SCHEMA))
.setTableType(conf.getString(FlinkOptions.TABLE_TYPE))
.setTableName(conf.getString(FlinkOptions.TABLE_NAME))
- .setDatabaseName(conf.getString(FlinkOptions.DATABASE_NAME))
Review Comment:
did you file a separate PR for fixing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374771378
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r985359891
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1052,13 +875,29 @@ public void dropIndex(List<MetadataPartitionType> partitionTypes) {
}
}
+ /**
+ * Performs Clustering for the workload stored in instant-time.
+ *
+ * @param clusteringInstantTime Clustering Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> cluster(String clusteringInstantTime) {
+ if (delegateToTableManagerService(config, ActionType.replacecommit)) {
+ throw new HoodieException(ActionType.replacecommit.name() + " delegate to table management service!");
Review Comment:
please align on the name
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseTableServiceClient.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+
+public abstract class BaseTableServiceClient<O> extends CommonHoodieClient {
Review Comment:
BaseHoodieTableServiceClient to align with BaseHoodieWriteClient
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
+
+ public static final String EXECUTE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/execute");
+ public static final String DELETE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/delete");
+
+ public static final String EXECUTE_CLEAN = String.format("%s/%s", BASE_URL, "clean/execute");
+ public static final String DELETE_CLEAN = String.format("%s/%s", BASE_URL, "clean/delete");
+
+ public static final String DATABASE_NAME_PARAM = "db_name";
+ public static final String TABLE_NAME_PARAM = "table_name";
+ public static final String BASEPATH_PARAM = "basepath";
+ public static final String INSTANT_PARAM = "instant";
+ public static final String USERNAME = "username";
+ public static final String CLUSTER = "cluster";
+ public static final String QUEUE = "queue";
+ public static final String RESOURCE = "resource";
+ public static final String PARALLELISM = "parallelism";
+ public static final String EXTRA_PARAMS = "extra_params";
+ public static final String EXECUTION_ENGINE = "execution_engine";
+
+ private final HoodieTableManagerConfig config;
+ private final HoodieTableMetaClient metaClient;
+ private final String host;
+ private final int port;
+ private final String basePath;
+ private final String dbName;
+ private final String tableName;
+
+ private static final Logger LOG = LogManager.getLogger(HoodieTableManagerClient.class);
+
+ public HoodieTableManagerClient(HoodieTableMetaClient metaClient, HoodieTableManagerConfig config) {
+ this.basePath = metaClient.getBasePathV2().toString();
+ this.dbName = metaClient.getTableConfig().getDatabaseName();
+ this.tableName = metaClient.getTableConfig().getTableName();
+ this.host = config.getTableManagerHost();
+ this.port = config.getTableManagerPort();
+ this.config = config;
+ this.metaClient = metaClient;
+ }
+
+ private String executeRequest(String requestPath, Map<String, String> queryParameters) throws IOException {
+ URIBuilder builder =
+ new URIBuilder().setHost(host).setPort(port).setPath(requestPath).setScheme("http");
Review Comment:
shouldn't the server URI be configured? including scheme, host, port, etc. so we don't hard code http here
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/RunsTableService.java:
##########
@@ -34,4 +35,12 @@ default boolean tableServicesEnabled(HoodieWriteConfig config) {
}
return enabled;
}
+
+ default boolean delegateToTableManagerService(HoodieWriteConfig config, ActionType actionType) {
Review Comment:
useTableManagerService suits better for this
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -270,51 +254,8 @@ public void preWrite(String instantTime, WriteOperationType writeOperationType,
// remove the async cleaning
}
- @Override
- protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
Review Comment:
please annotate in comments properly to explain the code removals
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -418,43 +311,13 @@ protected HoodieTable doInitTable(HoodieTableMetaClient metaClient, Option<Strin
// Initialize Metadata Table to make sure it's bootstrapped _before_ the operation,
// if it didn't exist before
// See https://issues.apache.org/jira/browse/HUDI-3343 for more details
- initializeMetadataTable(instantTime);
+ ((SparkRDDTableServiceClient) tableServiceClient).initializeMetadataTable(instantTime);
}
// Create a Hoodie table which encapsulated the commits and files visible
return HoodieSparkTable.create(config, (HoodieSparkEngineContext) context, metaClient);
}
- /**
- * Initialize the metadata table if needed. Creating the metadata table writer
- * will trigger the initial bootstrapping from the data table.
- *
- * @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
- */
- private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- }
- }
-
- // TODO : To enforce priority between table service and ingestion writer, use transactions here and invoke strategy
- private void completeTableService(TableServiceType tableServiceType, HoodieCommitMetadata metadata,
Review Comment:
ditto
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -125,7 +114,7 @@
* @param <K> Type of keys
* @param <O> Type of outputs
*/
-public abstract class BaseHoodieWriteClient<T extends HoodieRecordPayload, I, K, O> extends BaseHoodieClient
+public abstract class BaseHoodieWriteClient<T extends HoodieRecordPayload, I, K, O> extends CommonHoodieClient
Review Comment:
so the removed code is moved to the parent and TableServiceClient, right? for any removal from this class, please annotate the diff to clarify where it's moved to, it helps expedite review process.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/RunsTableService.java:
##########
@@ -34,4 +35,12 @@ default boolean tableServicesEnabled(HoodieWriteConfig config) {
}
return enabled;
}
+
+ default boolean delegateToTableManagerService(HoodieWriteConfig config, ActionType actionType) {
+ boolean supportsAction = config.getTableManagerConfig().isTableManagerSupportsAction(actionType);
+ if (supportsAction) {
+ LOG.warn(actionType.name() + " delegate to table manager service!");
+ }
+ return supportsAction;
Review Comment:
BaseTableServiceClient repeats this ? can we adopt the interface there?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1052,13 +875,29 @@ public void dropIndex(List<MetadataPartitionType> partitionTypes) {
}
}
+ /**
+ * Performs Clustering for the workload stored in instant-time.
+ *
+ * @param clusteringInstantTime Clustering Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> cluster(String clusteringInstantTime) {
+ if (delegateToTableManagerService(config, ActionType.replacecommit)) {
Review Comment:
so we call it Table Management Service or Table Manager Service? please align this with RFC doc. Or do we simply call it Table Manager?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
+
+ public static final String EXECUTE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/execute");
+ public static final String DELETE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/delete");
+
+ public static final String EXECUTE_CLEAN = String.format("%s/%s", BASE_URL, "clean/execute");
+ public static final String DELETE_CLEAN = String.format("%s/%s", BASE_URL, "clean/delete");
+
+ public static final String DATABASE_NAME_PARAM = "db_name";
+ public static final String TABLE_NAME_PARAM = "table_name";
+ public static final String BASEPATH_PARAM = "basepath";
+ public static final String INSTANT_PARAM = "instant";
+ public static final String USERNAME = "username";
+ public static final String CLUSTER = "cluster";
+ public static final String QUEUE = "queue";
+ public static final String RESOURCE = "resource";
+ public static final String PARALLELISM = "parallelism";
+ public static final String EXTRA_PARAMS = "extra_params";
+ public static final String EXECUTION_ENGINE = "execution_engine";
Review Comment:
use an ENUM to manage these text keys? or have a request composer to template and create the request payload?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseTableServiceClient.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+
+public abstract class BaseTableServiceClient<O> extends CommonHoodieClient {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected BaseTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig, HoodieMetrics metrics) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected boolean tableServicesEnabled(HoodieWriteConfig config) {
+ boolean enabled = config.areTableServicesEnabled();
+ if (!enabled) {
+ LOG.warn(String.format("Table services are disabled. Set `%s` to enable.", HoodieWriteConfig.TABLE_SERVICES_ENABLED));
+ }
+ return enabled;
+ }
+
+ protected boolean delegateToTableManagerService(HoodieWriteConfig config, ActionType actionType) {
Review Comment:
useTableManagerService() ? delegateToXXX does not imply boolean return
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseTableServiceClient.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+
+public abstract class BaseTableServiceClient<O> extends CommonHoodieClient {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected BaseTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig, HoodieMetrics metrics) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected boolean tableServicesEnabled(HoodieWriteConfig config) {
+ boolean enabled = config.areTableServicesEnabled();
+ if (!enabled) {
+ LOG.warn(String.format("Table services are disabled. Set `%s` to enable.", HoodieWriteConfig.TABLE_SERVICES_ENABLED));
+ }
+ return enabled;
+ }
+
+ protected boolean delegateToTableManagerService(HoodieWriteConfig config, ActionType actionType) {
+ boolean supportsAction = config.getTableManagerConfig().isTableManagerSupportsAction(actionType);
+ if (supportsAction) {
+ LOG.warn(actionType.name() + " delegate to table manager service!");
Review Comment:
the log statement looks redundant here. If really needs, make it debug
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2160,6 +2167,13 @@ public boolean isMetastoreEnabled() {
return metastoreConfig.enableMetastore();
}
+ /**
+ * Table Manager configs.
+ */
+ public boolean isTableManagerEnabled() {
+ return tableManagerConfig.enableTableManager();
Review Comment:
again let's make sure the name is finalized: Table Management Service, or Table Manager Service, or Table Manager?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
+
+ public static final String EXECUTE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/execute");
+ public static final String DELETE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/delete");
+
+ public static final String EXECUTE_CLEAN = String.format("%s/%s", BASE_URL, "clean/execute");
+ public static final String DELETE_CLEAN = String.format("%s/%s", BASE_URL, "clean/delete");
+
+ public static final String DATABASE_NAME_PARAM = "db_name";
+ public static final String TABLE_NAME_PARAM = "table_name";
+ public static final String BASEPATH_PARAM = "basepath";
+ public static final String INSTANT_PARAM = "instant";
+ public static final String USERNAME = "username";
+ public static final String CLUSTER = "cluster";
+ public static final String QUEUE = "queue";
+ public static final String RESOURCE = "resource";
+ public static final String PARALLELISM = "parallelism";
+ public static final String EXTRA_PARAMS = "extra_params";
+ public static final String EXECUTION_ENGINE = "execution_engine";
+
+ private final HoodieTableManagerConfig config;
+ private final HoodieTableMetaClient metaClient;
+ private final String host;
+ private final int port;
+ private final String basePath;
+ private final String dbName;
+ private final String tableName;
+
+ private static final Logger LOG = LogManager.getLogger(HoodieTableManagerClient.class);
+
+ public HoodieTableManagerClient(HoodieTableMetaClient metaClient, HoodieTableManagerConfig config) {
+ this.basePath = metaClient.getBasePathV2().toString();
+ this.dbName = metaClient.getTableConfig().getDatabaseName();
+ this.tableName = metaClient.getTableConfig().getTableName();
+ this.host = config.getTableManagerHost();
+ this.port = config.getTableManagerPort();
+ this.config = config;
+ this.metaClient = metaClient;
+ }
+
+ private String executeRequest(String requestPath, Map<String, String> queryParameters) throws IOException {
+ URIBuilder builder =
+ new URIBuilder().setHost(host).setPort(port).setPath(requestPath).setScheme("http");
+ queryParameters.forEach(builder::addParameter);
+
+ String url = builder.toString();
+ LOG.info("Sending request to table management service : (" + url + ")");
+ Response response;
+ int timeout = this.config.getConnectionTimeout() * 1000; // msec
+ int requestRetryLimit = config.getConnectionRetryLimit();
+ int retry = 0;
+
+ while (retry < requestRetryLimit) {
+ try {
+ response = Request.Get(url).connectTimeout(timeout).socketTimeout(timeout).execute();
+ return response.returnContent().asString();
+ } catch (IOException e) {
+ retry++;
+ LOG.warn(String.format("Failed request to server %s, will retry for %d times", url, requestRetryLimit - retry), e);
+ if (requestRetryLimit == retry) {
+ throw e;
+ }
+ }
+
+ try {
+ TimeUnit.SECONDS.sleep(config.getConnectionRetryDelay());
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ }
+
+ throw new IOException(String.format("Failed request to table management service %s after retry %d times", url, requestRetryLimit));
+ }
+
+ private Map<String, String> getParamsWithAdditionalParams(String[] paramNames, String[] paramVals) {
Review Comment:
why is it called "AdditionalParams" ? the method creates a new map and put everything inside. it's just putting key vals together. The name makes it sounds like associating with the EXTRA_PARAMS. let's name it properly.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
+
+ public static final String EXECUTE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/execute");
+ public static final String DELETE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/delete");
+
+ public static final String EXECUTE_CLEAN = String.format("%s/%s", BASE_URL, "clean/execute");
+ public static final String DELETE_CLEAN = String.format("%s/%s", BASE_URL, "clean/delete");
+
+ public static final String DATABASE_NAME_PARAM = "db_name";
+ public static final String TABLE_NAME_PARAM = "table_name";
+ public static final String BASEPATH_PARAM = "basepath";
+ public static final String INSTANT_PARAM = "instant";
+ public static final String USERNAME = "username";
+ public static final String CLUSTER = "cluster";
+ public static final String QUEUE = "queue";
+ public static final String RESOURCE = "resource";
+ public static final String PARALLELISM = "parallelism";
+ public static final String EXTRA_PARAMS = "extra_params";
+ public static final String EXECUTION_ENGINE = "execution_engine";
+
+ private final HoodieTableManagerConfig config;
+ private final HoodieTableMetaClient metaClient;
+ private final String host;
+ private final int port;
+ private final String basePath;
+ private final String dbName;
+ private final String tableName;
+
+ private static final Logger LOG = LogManager.getLogger(HoodieTableManagerClient.class);
+
+ public HoodieTableManagerClient(HoodieTableMetaClient metaClient, HoodieTableManagerConfig config) {
+ this.basePath = metaClient.getBasePathV2().toString();
+ this.dbName = metaClient.getTableConfig().getDatabaseName();
+ this.tableName = metaClient.getTableConfig().getTableName();
+ this.host = config.getTableManagerHost();
+ this.port = config.getTableManagerPort();
+ this.config = config;
+ this.metaClient = metaClient;
+ }
+
+ private String executeRequest(String requestPath, Map<String, String> queryParameters) throws IOException {
+ URIBuilder builder =
+ new URIBuilder().setHost(host).setPort(port).setPath(requestPath).setScheme("http");
+ queryParameters.forEach(builder::addParameter);
+
+ String url = builder.toString();
+ LOG.info("Sending request to table management service : (" + url + ")");
+ Response response;
+ int timeout = this.config.getConnectionTimeout() * 1000; // msec
+ int requestRetryLimit = config.getConnectionRetryLimit();
+ int retry = 0;
+
+ while (retry < requestRetryLimit) {
Review Comment:
can we make use org.apache.hudi.common.util.RetryHelper here? also pls enhance the helper wherever applicable
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
Review Comment:
"execute" sounds like a synchronous process, while we are actually requesting (async) right? it's more accurate with "request". Same problem with "delete". shouldn't it be "cancel"?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -159,9 +162,19 @@ protected Option<HoodieCleanerPlan> requestClean(String startCleanTime) {
LOG.error("Got exception when saving cleaner requested file", e);
throw new HoodieIOException(e.getMessage(), e);
}
- return Option.of(cleanerPlan);
+ option = Option.of(cleanerPlan);
}
- return Option.empty();
+
+ if (config.getTableManagerConfig().isTableManagerSupportsAction(ActionType.clean)) {
+ delegateCleanExecutionToTableManager();
+ }
Review Comment:
this reads like "as long as table manager supports this action, we delegate it". but internally it checks enabling or not. we should avoid this confusion.
We should also make the logic flow cleanly separated: can you make the delegation return Option<HoodieCleanerPlan> and fully separated from the existing flow? if to delegate, then let the delegate do everything. This applies to other executor changes.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableManagerConfig.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the HUDI Table Management Service.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Management Service Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Management Service.")
+public class HoodieTableManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_MANAGEMENT_SERVICE_PREFIX = "hoodie.table.management.service";
Review Comment:
name alignment, applies to all configs below
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/table/manager/HoodieTableManagerClient.java:
##########
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.table.manager;
+
+import org.apache.hudi.common.config.HoodieTableManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Client which send the table service instants to the table management service.
+ */
+public class HoodieTableManagerClient {
+
+ private static final String BASE_URL = "/v1/hoodie/service";
+
+ public static final String REGISTER_ENDPOINT = String.format("%s/%s", BASE_URL, "register");
+
+ public static final String EXECUTE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/execute");
+ public static final String DELETE_COMPACTION = String.format("%s/%s", BASE_URL, "compact/delete");
+
+ public static final String EXECUTE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/execute");
+ public static final String DELETE_CLUSTERING = String.format("%s/%s", BASE_URL, "cluster/delete");
+
+ public static final String EXECUTE_CLEAN = String.format("%s/%s", BASE_URL, "clean/execute");
+ public static final String DELETE_CLEAN = String.format("%s/%s", BASE_URL, "clean/delete");
Review Comment:
the endpoint url pattern looks more like RPC rather than REST. if the design decides to go with REST, then i think the action itself like execute/delete should be part of the payload. If go with RPC, this is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r995431475
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1052,13 +875,29 @@ public void dropIndex(List<MetadataPartitionType> partitionTypes) {
}
}
+ /**
+ * Performs Clustering for the workload stored in instant-time.
+ *
+ * @param clusteringInstantTime Clustering Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> cluster(String clusteringInstantTime) {
+ if (delegateToTableManagerService(config, ActionType.replacecommit)) {
Review Comment:
Already update.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -1052,13 +875,29 @@ public void dropIndex(List<MetadataPartitionType> partitionTypes) {
}
}
+ /**
+ * Performs Clustering for the workload stored in instant-time.
+ *
+ * @param clusteringInstantTime Clustering Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> cluster(String clusteringInstantTime) {
+ if (delegateToTableManagerService(config, ActionType.replacecommit)) {
+ throw new HoodieException(ActionType.replacecommit.name() + " delegate to table management service!");
Review Comment:
Already update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1297002813
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1306769066
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 0da5a658baa0616cfb0666fe6a1d0d916191d274 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855)
* f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253172264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 195a76d8e8369bce14d09ac2b46bfb293eb858ae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374497670
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1375039783
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1373391935
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1375834595
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1071511854
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -283,44 +270,14 @@ public void preWrite(String instantTime, WriteOperationType writeOperationType,
@Override
protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
- try (HoodieBackedTableMetadataWriter metadataWriter = initMetadataWriter()) {
- metadataWriter.update(metadata, instantTime, getHoodieTable().isTableServiceAction(actionType, instantTime));
- } catch (Exception e) {
- throw new HoodieException("Failed to update metadata", e);
- }
- }
-
- /**
- * Initialize the table metadata writer, for e.g, bootstrap the metadata table
- * from the filesystem if it does not exist.
- */
- private HoodieBackedTableMetadataWriter initMetadataWriter() {
- return (HoodieBackedTableMetadataWriter) FlinkHoodieBackedTableMetadataWriter.create(
- FlinkClientUtil.getHadoopConf(), this.config, HoodieFlinkEngineContext.DEFAULT);
+ tableServiceClient.writeTableMetadata(table, instantTime, actionType, metadata);
}
/**
* Initialized the metadata table on start up, should only be called once on driver.
*/
public void initMetadataTable() {
- HoodieFlinkTable<?> table = getHoodieTable();
- if (config.isMetadataTableEnabled()) {
- // initialize the metadata table path
- // guard the metadata writer with concurrent lock
- try {
- this.txnManager.getLockManager().lock();
- initMetadataWriter().close();
- } catch (Exception e) {
- throw new HoodieException("Failed to initialize metadata table", e);
- } finally {
- this.txnManager.getLockManager().unlock();
- }
- // clean the obsolete index stats
- table.deleteMetadataIndexIfNecessary();
- } else {
- // delete the metadata table if it was enabled but is now disabled
- table.maybeDeleteMetadataTable();
- }
+ ((HoodieFlinkTableServiceClient<T>) tableServiceClient).initMetadataTable();
Review Comment:
@yuzhaojing is there a jira?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058314660
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/HoodieTableServiceManagerClient.java:
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.common.config.HoodieTableServiceManagerConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.RetryHelper;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieRemoteException;
+
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Client which send the table service instants to the table service manager.
+ */
+public class HoodieTableServiceManagerClient {
+
+ /**
+ * Rollback commands, that trigger a specific handling for rollback.
+ */
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1364068271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
}, {
"hash" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939",
"triggerID" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"triggerType" : "PUSH"
}, {
"hash" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13943",
"triggerID" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* 24f7f28bb6f5d124e250afae9e5a93b51cb0bd07 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13943)
* 097828528e3529f0590d38be30e27e2787a64fa2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058027049
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieTableServiceManagerConfig.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.config;
+
+import org.apache.hudi.common.model.ActionType;
+
+import javax.annotation.concurrent.Immutable;
+
+import java.util.Properties;
+
+/**
+ * Configurations used by the Hudi Table Service Manager.
+ */
+@Immutable
+@ConfigClassProperty(name = "Table Service Manager Configs",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ description = "Configurations used by the Hudi Table Service Manager.")
+public class HoodieTableServiceManagerConfig extends HoodieConfig {
+
+ public static final String TABLE_SERVICE_MANAGER_PREFIX = "hoodie.table.service.manager";
+
+ public static final ConfigProperty<Boolean> TABLE_SERVICE_MANAGER_ENABLE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".enable")
+ .defaultValue(false)
+ .withDocumentation("Use table manager service to execute table service");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_URIS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".uris")
+ .defaultValue("http://localhost:9091")
+ .withDocumentation("Table service manager uris");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_ACTIONS = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".actions")
+ .defaultValue("")
+ .withDocumentation("Which action deploy on table service manager such as compaction:clean, default null");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_USERNAME = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.username")
+ .defaultValue("default")
+ .withDocumentation("The user name to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_QUEUE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.queue")
+ .defaultValue("default")
+ .withDocumentation("The queue to deploy for table service of this table");
+
+ public static final ConfigProperty<String> TABLE_SERVICE_MANAGER_DEPLOY_RESOURCE = ConfigProperty
+ .key(TABLE_SERVICE_MANAGER_PREFIX + ".deploy.resource")
+ .defaultValue("4g:4g")
+ .withDocumentation("The resource to deploy for table service of this table, default driver 4g, executor 4g");
Review Comment:
Spark engine and flink engine can share this configuration, so I don't think there is any need to make a distinction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366814215
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64ecea100e226b7fd539cab05c03bc9902e36db1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015)
* c20aa589730546c0c7bb82969c92aa6d364af101 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366811068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 64ecea100e226b7fd539cab05c03bc9902e36db1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015)
* c20aa589730546c0c7bb82969c92aa6d364af101 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1058026706
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -84,6 +84,12 @@ private FlinkOptions() {
// Common Options
// ------------------------------------------------------------------------
+ public static final ConfigOption<String> DATABASE_NAME = ConfigOptions
+ .key(HoodieTableConfig.DATABASE_NAME.key())
+ .stringType()
+ .noDefaultValue()
+ .withDescription("Database name to register to Hive metastore");
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1371927620
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20aa589730546c0c7bb82969c92aa6d364af101 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033)
* 28aacdc92d0f7027e03cb2cd8891b19b48e47da0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1371933207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114",
"triggerID" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20aa589730546c0c7bb82969c92aa6d364af101 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033)
* 28aacdc92d0f7027e03cb2cd8891b19b48e47da0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374047811
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152)
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1378411330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14190",
"triggerID" : "1375486285",
"triggerType" : "MANUAL"
}, {
"hash" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14234",
"triggerID" : "e4701cd7f6ba883bca09d1c99b32135bf827c58d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e4701cd7f6ba883bca09d1c99b32135bf827c58d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14234)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1373193285
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1375186469
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180",
"triggerID" : "1375039153",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14172) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14171) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14180)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1373392309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144",
"triggerID" : "1373391935",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14135) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1374499289
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14144",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14152",
"triggerID" : "1374481061",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1384265462
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f27b8bccfd4bfd64c142c94fe50660ac9644ff2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1f27b8bccfd4bfd64c142c94fe50660ac9644ff2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1364061275
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
}, {
"hash" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939",
"triggerID" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"triggerType" : "PUSH"
}, {
"hash" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13943",
"triggerID" : "24f7f28bb6f5d124e250afae9e5a93b51cb0bd07",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* a614791ea45854805a892bf53ec99f0dcdad30c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939)
* 24f7f28bb6f5d124e250afae9e5a93b51cb0bd07 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13943)
* 097828528e3529f0590d38be30e27e2787a64fa2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1363692253
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12879",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
}, {
"hash" : "27ae850689d655b0054d4a106c10b418ff662fda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12949",
"triggerID" : "27ae850689d655b0054d4a106c10b418ff662fda",
"triggerType" : "PUSH"
}, {
"hash" : "46f647dae80082ea855489fd86c42b17368af005",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46f647dae80082ea855489fd86c42b17368af005",
"triggerType" : "PUSH"
}, {
"hash" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916",
"triggerID" : "f7e140b4aff4e87ca18e8390faa04e1c72b92378",
"triggerType" : "PUSH"
}, {
"hash" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939",
"triggerID" : "a614791ea45854805a892bf53ec99f0dcdad30c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 46f647dae80082ea855489fd86c42b17368af005 UNKNOWN
* f7e140b4aff4e87ca18e8390faa04e1c72b92378 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13916)
* a614791ea45854805a892bf53ec99f0dcdad30c6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13939)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1364937919
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366150250
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994",
"triggerID" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 022edcdb913eff2db16c824095f5e836a81c3b44 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994)
* bd837f2febbd7c102305f91f8e85532782d29467 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366146554
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13944",
"triggerID" : "097828528e3529f0590d38be30e27e2787a64fa2",
"triggerType" : "PUSH"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "097828528e3529f0590d38be30e27e2787a64fa2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13970",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994",
"triggerID" : "022edcdb913eff2db16c824095f5e836a81c3b44",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 022edcdb913eff2db16c824095f5e836a81c3b44 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13994)
* bd837f2febbd7c102305f91f8e85532782d29467 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366643203
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 64ecea100e226b7fd539cab05c03bc9902e36db1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1366895616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c20aa589730546c0c7bb82969c92aa6d364af101 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1278657991
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 011023c6bb6d766a7b3098312ff1352ebe05136f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196)
* d4408906bcc13c55f6350f35c168be32d978daf6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1278563566
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* 011023c6bb6d766a7b3098312ff1352ebe05136f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1253174691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 195a76d8e8369bce14d09ac2b46bfb293eb858ae Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1297014210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* d4408906bcc13c55f6350f35c168be32d978daf6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204)
* df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1307190831
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
}, {
"hash" : "36e29156d84985f88e46463b6177e3e751bbe013",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36e29156d84985f88e46463b6177e3e751bbe013",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869)
* 36e29156d84985f88e46463b6177e3e751bbe013 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table management service
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1307077688
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11544",
"triggerID" : "195a76d8e8369bce14d09ac2b46bfb293eb858ae",
"triggerType" : "PUSH"
}, {
"hash" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11547",
"triggerID" : "149e8f3c61e1221c4659da00a4168faeed1174dd",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"triggerType" : "PUSH"
}, {
"hash" : "78f3167ee6aba247ef55dcfacf8c2be603d621be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11549",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12181",
"triggerID" : "b35c4e70a6564f5524b1e662142b51d419ccaeff",
"triggerType" : "PUSH"
}, {
"hash" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ed7915b066588e620163ecd0abafc6c61fd587a",
"triggerType" : "PUSH"
}, {
"hash" : "175998dab136062e0244c857fb3ef8982ceeada7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12185",
"triggerID" : "175998dab136062e0244c857fb3ef8982ceeada7",
"triggerType" : "PUSH"
}, {
"hash" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12196",
"triggerID" : "011023c6bb6d766a7b3098312ff1352ebe05136f",
"triggerType" : "PUSH"
}, {
"hash" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12204",
"triggerID" : "d4408906bcc13c55f6350f35c168be32d978daf6",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12686",
"triggerID" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"triggerType" : "PUSH"
}, {
"hash" : "df2eb52b0dcdc96a39fa3270e9b2f3ee16409a1b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12689",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12690",
"triggerID" : "7cf07c0681d32e8831af9f39168695d5540af636",
"triggerType" : "PUSH"
}, {
"hash" : "7cf07c0681d32e8831af9f39168695d5540af636",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12850",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12855",
"triggerID" : "0da5a658baa0616cfb0666fe6a1d0d916191d274",
"triggerType" : "PUSH"
}, {
"hash" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869",
"triggerID" : "f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4ed7915b066588e620163ecd0abafc6c61fd587a UNKNOWN
* f8eb50f7eb4d89ac18a413aec5dce1b1a4f7aef0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12869)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6732:
URL: https://github.com/apache/hudi/pull/6732#issuecomment-1372400330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13996",
"triggerID" : "bd837f2febbd7c102305f91f8e85532782d29467",
"triggerType" : "PUSH"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1305495583",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "bd837f2febbd7c102305f91f8e85532782d29467",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14003",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1254627373",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1297002813",
"triggerType" : "MANUAL"
}, {
"hash" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14015",
"triggerID" : "64ecea100e226b7fd539cab05c03bc9902e36db1",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14020",
"triggerID" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"triggerType" : "PUSH"
}, {
"hash" : "c20aa589730546c0c7bb82969c92aa6d364af101",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14033",
"triggerID" : "1367108432",
"triggerType" : "MANUAL"
}, {
"hash" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14114",
"triggerID" : "28aacdc92d0f7027e03cb2cd8891b19b48e47da0",
"triggerType" : "PUSH"
}, {
"hash" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14121",
"triggerID" : "dff9924f25bf457dc12cdd7a7a11b923ad08c3b7",
"triggerType" : "PUSH"
}, {
"hash" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122",
"triggerID" : "af728e3eeb1fd694eba037bc9a48869831ddb053",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af728e3eeb1fd694eba037bc9a48869831ddb053 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14122)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6732: [HUDI-4148] Add client for hudi table service manager
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6732:
URL: https://github.com/apache/hudi/pull/6732#discussion_r1060687626
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieClient.java:
##########
@@ -85,6 +91,8 @@ protected BaseHoodieClient(HoodieEngineContext context, HoodieWriteConfig client
public void close() {
stopEmbeddedServerView(true);
this.context.setJobStatus("", "");
+ this.heartbeatClient.stop();
+ this.txnManager.close();
Review Comment:
remark: this is common to all clients, so moved here.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -0,0 +1,840 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class BaseHoodieTableServiceClient<O> extends BaseHoodieClient implements RunsTableService {
Review Comment:
remark: logic for table services like clean, cluster, compact, archive, rollback were moved here from base write client
##########
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/client/HoodieJavaTableServiceClient.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieNotSupportedException;
+import org.apache.hudi.table.HoodieJavaTable;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.hadoop.conf.Configuration;
+
+import java.util.List;
+import java.util.Map;
+
+public class HoodieJavaTableServiceClient extends BaseHoodieTableServiceClient<List<WriteStatus>> {
Review Comment:
remark: the table services' logic were replicated as per java write client's implementation
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -0,0 +1,840 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class BaseHoodieTableServiceClient<O> extends BaseHoodieClient implements RunsTableService {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+
+ protected BaseHoodieTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected void startAsyncCleanerService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncCleanerService == null) {
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(writeClient);
+ } else {
+ this.asyncCleanerService.start(null);
+ }
+ }
+
+ protected void startAsyncArchiveService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncArchiveService == null) {
+ this.asyncArchiveService = AsyncArchiveService.startAsyncArchiveIfEnabled(writeClient);
+ } else {
+ this.asyncArchiveService.start(null);
+ }
+ }
+
+ protected void asyncClean() {
+ AsyncCleanerService.waitForCompletion(asyncCleanerService);
+ }
+
+ protected void asyncArchive() {
+ AsyncArchiveService.waitForCompletion(asyncArchiveService);
+ }
+
+ protected void setTableServiceTimer(WriteOperationType operationType) {
+ switch (operationType) {
+ case CLUSTER:
+ clusteringTimer = metrics.getClusteringCtx();
+ break;
+ case COMPACT:
+ compactionTimer = metrics.getCompactionCtx();
+ break;
+ case LOG_COMPACT:
+ logCompactionTimer = metrics.getLogCompactionCtx();
+ break;
+ default:
+ }
+ }
+
+ /**
+ * Performs a compaction operation on a table, serially before or after an insert/upsert action.
+ * Scheduling and execution is done inline.
+ */
+ protected Option<String> inlineCompaction(Option<Map<String, String>> extraMetadata) {
+ Option<String> compactionInstantTimeOpt = inlineScheduleCompaction(extraMetadata);
+ compactionInstantTimeOpt.ifPresent(compactInstantTime -> {
+ // inline compaction should auto commit as the user is never given control
+ compact(compactInstantTime, true);
+ });
+ return compactionInstantTimeOpt;
+ }
+
+ private void inlineCompaction(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.compaction)) {
+ scheduleCompaction(extraMetadata);
+ } else {
+ runAnyPendingCompactions(table);
+ inlineCompaction(extraMetadata);
+ }
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Log Compaction for the workload stored in instant-time.s
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected HoodieWriteMetadata<O> logCompact(String compactionInstantTime, boolean shouldComplete) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+ /**
+ * Performs a log compaction operation on a table, serially before or after an insert/upsert action.
+ */
+ protected Option<String> inlineLogCompact(Option<Map<String, String>> extraMetadata) {
+ Option<String> logCompactionInstantTimeOpt = scheduleLogCompaction(extraMetadata);
+ logCompactionInstantTimeOpt.ifPresent(logCompactInstantTime -> {
+ // inline log compaction should auto commit as the user is never given control
+ logCompact(logCompactInstantTime, true);
+ });
+ return logCompactionInstantTimeOpt;
+ }
+
+ protected void runAnyPendingCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight compaction at instant " + instant);
+ compact(instant.getTimestamp(), true);
+ });
+ }
+
+ protected void runAnyPendingLogCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingLogCompactionTimeline().getInstantsAsStream()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight log compaction at instant " + instant);
+ logCompact(instant.getTimestamp(), true);
+ });
+ }
+
+ /***
+ * Schedules compaction inline.
+ * @param extraMetadata extrametada to be used.
+ * @return compaction instant if scheduled.
+ */
+ protected Option<String> inlineScheduleCompaction(Option<Map<String, String>> extraMetadata) {
+ return scheduleCompaction(extraMetadata);
+ }
+
+ /**
+ * Schedules a new compaction instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected abstract HoodieWriteMetadata<O> compact(String compactionInstantTime, boolean shouldComplete);
+
+ /**
+ * Commit a compaction operation. Allow passing additional meta-data to be stored in commit instant file.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @param metadata All the metadata that gets stored along with a commit
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public abstract void commitCompaction(String compactionInstantTime, HoodieCommitMetadata metadata,
+ Option<Map<String, String>> extraMetadata);
+
+ /**
+ * Commit Compaction and track metrics.
+ */
+ protected abstract void completeCompaction(HoodieCommitMetadata metadata, HoodieTable table, String compactionCommitTime);
+
+ /**
+ * Schedules a new log compaction instant.
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleLogCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleLogCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new log compaction instant with passed-in instant time.
+ * @param instantTime Log Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleLogCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.LOG_COMPACT).isPresent();
+ }
+
+ /**
+ * Performs Log Compaction for the workload stored in instant-time.
+ *
+ * @param logCompactionInstantTime Log Compaction Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> logCompact(String logCompactionInstantTime) {
+ return logCompact(logCompactionInstantTime, config.shouldAutoCommit());
+ }
+
+ /**
+ * Commit Log Compaction and track metrics.
+ */
+ protected void completeLogCompaction(HoodieCommitMetadata metadata, HoodieTable table, String logCompactionCommitTime) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+
+ /**
+ * Schedules a new compaction instant with passed-in instant time.
+ *
+ * @param instantTime Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.COMPACT).isPresent();
+ }
+
+ /**
+ * Schedules a new clustering instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleClustering(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleClusteringAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new clustering instant with passed-in instant time.
+ *
+ * @param instantTime clustering Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleClusteringAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLUSTER).isPresent();
+ }
+
+ /**
+ * Schedules a new cleaning instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected Option<String> scheduleCleaning(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCleaningAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new cleaning instant with passed-in instant time.
+ *
+ * @param instantTime cleaning Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected boolean scheduleCleaningAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLEAN).isPresent();
+ }
+
+ /**
+ * Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.
+ *
+ * @param clusteringInstant Clustering Instant Time
+ * @return Collection of Write Status
+ */
+ public abstract HoodieWriteMetadata<O> cluster(String clusteringInstant, boolean shouldComplete);
+
+ protected void runTableServicesInline(HoodieTable table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
+ if (!tableServicesEnabled(config)) {
+ return;
+ }
+
+ if (!config.areAnyTableServicesExecutedInline() && !config.areAnyTableServicesScheduledInline()) {
+ return;
+ }
+
+ if (config.isMetadataTableEnabled()) {
+ table.getHoodieView().sync();
+ }
+ // Do an inline compaction if enabled
+ if (config.inlineCompactionEnabled()) {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
+ inlineCompaction(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().empty()) {
+ // proceed only if there are no pending compactions
+ metadata.addMetadata(HoodieCompactionConfig.SCHEDULE_INLINE_COMPACT.key(), "true");
+ inlineScheduleCompaction(extraMetadata);
+ }
+
+ // Do an inline log compaction if enabled
+ if (config.inlineLogCompactionEnabled()) {
+ runAnyPendingLogCompactions(table);
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "true");
+ inlineLogCompact(extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "false");
+ }
+
+ // Do an inline clustering if enabled
+ if (config.inlineClusteringEnabled()) {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "true");
+ inlineClustering(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineClusteringEnabled() && config.scheduleInlineClustering()
+ && table.getActiveTimeline().filterPendingReplaceTimeline().empty()) {
+ // proceed only if there are no pending clustering
+ metadata.addMetadata(HoodieClusteringConfig.SCHEDULE_INLINE_CLUSTERING.key(), "true");
+ inlineScheduleClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedule table services such as clustering, compaction & cleaning.
+ *
+ * @param extraMetadata Metadata to pass onto the scheduled service instant
+ * @param tableServiceType Type of table service to schedule
+ * @return
+ */
+ public Option<String> scheduleTableService(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ // A lock is required to guard against race conditions between an on-going writer and scheduling a table service.
+ final Option<HoodieInstant> inflightInstant = Option.of(new HoodieInstant(HoodieInstant.State.REQUESTED,
+ tableServiceType.getAction(), instantTime));
+ try {
+ this.txnManager.beginTransaction(inflightInstant, Option.empty());
+ LOG.info("Scheduling table service " + tableServiceType);
+ return scheduleTableServiceInternal(instantTime, extraMetadata, tableServiceType);
+ } finally {
+ this.txnManager.endTransaction(inflightInstant);
+ }
+ }
+
+ protected Option<String> scheduleTableServiceInternal(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ if (!tableServicesEnabled(config)) {
+ return Option.empty();
+ }
+
+ Option<String> option = Option.empty();
+ HoodieTable<?, ?, ?, ?> table = createTable(config, hadoopConf);
+
+ switch (tableServiceType) {
+ case ARCHIVE:
+ LOG.info("Scheduling archiving is not supported. Skipping.");
+ break;
+ case CLUSTER:
+ LOG.info("Scheduling clustering at instant time :" + instantTime);
+ Option<HoodieClusteringPlan> clusteringPlan = table
+ .scheduleClustering(context, instantTime, extraMetadata);
+ option = clusteringPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case COMPACT:
+ LOG.info("Scheduling compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> compactionPlan = table
+ .scheduleCompaction(context, instantTime, extraMetadata);
+ option = compactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case LOG_COMPACT:
+ LOG.info("Scheduling log compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> logCompactionPlan = table
+ .scheduleLogCompaction(context, instantTime, extraMetadata);
+ option = logCompactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case CLEAN:
+ LOG.info("Scheduling cleaning at instant time :" + instantTime);
+ Option<HoodieCleanerPlan> cleanerPlan = table
+ .scheduleCleaning(context, instantTime, extraMetadata);
+ option = cleanerPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ default:
+ throw new IllegalArgumentException("Invalid TableService " + tableServiceType);
+ }
+
+ Option<String> instantRange = delegateToTableServiceManager(tableServiceType, table);
+ if (instantRange.isPresent()) {
+ LOG.info("Delegate instant [" + instantRange.get() + "] to table service manager");
+ }
+
+ return option;
+ }
+
+ protected abstract HoodieTable<?, ?, ?, ?> createTable(HoodieWriteConfig config, Configuration hadoopConf);
+
+ /**
+ * Executes a clustering plan on a table, serially before or after an insert/upsert action.
+ * Schedules and executes clustering inline.
+ */
+ protected Option<String> inlineClustering(Option<Map<String, String>> extraMetadata) {
+ Option<String> clusteringInstantOpt = inlineScheduleClustering(extraMetadata);
+ clusteringInstantOpt.ifPresent(clusteringInstant -> {
+ // inline cluster should auto commit as the user is never given control
+ cluster(clusteringInstant, true);
+ });
+ return clusteringInstantOpt;
+ }
+
+ private void inlineClustering(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.replacecommit)) {
+ scheduleClustering(extraMetadata);
+ } else {
+ runAnyPendingClustering(table);
+ inlineClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedules clustering inline.
+ *
+ * @param extraMetadata extrametadata to use.
+ * @return clustering instant if scheduled.
+ */
+ protected Option<String> inlineScheduleClustering(Option<Map<String, String>> extraMetadata) {
+ return scheduleClustering(extraMetadata);
+ }
+
+ protected void runAnyPendingClustering(HoodieTable table) {
+ table.getActiveTimeline().filterPendingReplaceTimeline().getInstants().forEach(instant -> {
+ Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(table.getMetaClient(), instant);
+ if (instantPlan.isPresent()) {
+ LOG.info("Running pending clustering at instant " + instantPlan.get().getLeft());
+ cluster(instant.getTimestamp(), true);
+ }
+ });
+ }
+
+ /**
+ * Finalize Write operation.
+ *
+ * @param table HoodieTable
+ * @param instantTime Instant Time
+ * @param stats Hoodie Write Stat
+ */
+ protected void finalizeWrite(HoodieTable table, String instantTime, List<HoodieWriteStat> stats) {
+ try {
+ final Timer.Context finalizeCtx = metrics.getFinalizeCtx();
+ table.finalizeWrite(context, instantTime, stats);
+ if (finalizeCtx != null) {
+ Option<Long> durationInMs = Option.of(metrics.getDurationInMs(finalizeCtx.stop()));
+ durationInMs.ifPresent(duration -> {
+ LOG.info("Finalize write elapsed time (milliseconds): " + duration);
+ metrics.updateFinalizeWriteMetrics(duration, stats.size());
+ });
+ }
+ } catch (HoodieIOException ioe) {
+ throw new HoodieCommitException("Failed to complete commit " + instantTime + " due to finalize errors.", ioe);
+ }
+ }
+
+ /**
+ * Write the HoodieCommitMetadata to metadata table if available.
+ *
+ * @param table {@link HoodieTable} of interest.
+ * @param instantTime instant time of the commit.
+ * @param actionType action type of the commit.
+ * @param metadata instance of {@link HoodieCommitMetadata}.
+ */
+ protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
+ context.setJobStatus(this.getClass().getSimpleName(), "Committing to metadata table: " + config.getTableName());
+ table.getMetadataWriter(instantTime).ifPresent(w -> ((HoodieTableMetadataWriter) w).update(metadata, instantTime,
+ table.isTableServiceAction(actionType, instantTime)));
Review Comment:
this method in table service client should always assert isTableServiceAction() is true. we should do a ValidationUtils.checkArgument at the beginning of this method
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -0,0 +1,840 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class BaseHoodieTableServiceClient<O> extends BaseHoodieClient implements RunsTableService {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+
+ protected BaseHoodieTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected void startAsyncCleanerService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncCleanerService == null) {
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(writeClient);
+ } else {
+ this.asyncCleanerService.start(null);
+ }
+ }
+
+ protected void startAsyncArchiveService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncArchiveService == null) {
+ this.asyncArchiveService = AsyncArchiveService.startAsyncArchiveIfEnabled(writeClient);
+ } else {
+ this.asyncArchiveService.start(null);
+ }
+ }
+
+ protected void asyncClean() {
+ AsyncCleanerService.waitForCompletion(asyncCleanerService);
+ }
+
+ protected void asyncArchive() {
+ AsyncArchiveService.waitForCompletion(asyncArchiveService);
+ }
+
+ protected void setTableServiceTimer(WriteOperationType operationType) {
+ switch (operationType) {
+ case CLUSTER:
+ clusteringTimer = metrics.getClusteringCtx();
+ break;
+ case COMPACT:
+ compactionTimer = metrics.getCompactionCtx();
+ break;
+ case LOG_COMPACT:
+ logCompactionTimer = metrics.getLogCompactionCtx();
+ break;
+ default:
+ }
+ }
+
+ /**
+ * Performs a compaction operation on a table, serially before or after an insert/upsert action.
+ * Scheduling and execution is done inline.
+ */
+ protected Option<String> inlineCompaction(Option<Map<String, String>> extraMetadata) {
+ Option<String> compactionInstantTimeOpt = inlineScheduleCompaction(extraMetadata);
+ compactionInstantTimeOpt.ifPresent(compactInstantTime -> {
+ // inline compaction should auto commit as the user is never given control
+ compact(compactInstantTime, true);
+ });
+ return compactionInstantTimeOpt;
+ }
+
+ private void inlineCompaction(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.compaction)) {
+ scheduleCompaction(extraMetadata);
+ } else {
+ runAnyPendingCompactions(table);
+ inlineCompaction(extraMetadata);
+ }
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Log Compaction for the workload stored in instant-time.s
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected HoodieWriteMetadata<O> logCompact(String compactionInstantTime, boolean shouldComplete) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+ /**
+ * Performs a log compaction operation on a table, serially before or after an insert/upsert action.
+ */
+ protected Option<String> inlineLogCompact(Option<Map<String, String>> extraMetadata) {
+ Option<String> logCompactionInstantTimeOpt = scheduleLogCompaction(extraMetadata);
+ logCompactionInstantTimeOpt.ifPresent(logCompactInstantTime -> {
+ // inline log compaction should auto commit as the user is never given control
+ logCompact(logCompactInstantTime, true);
+ });
+ return logCompactionInstantTimeOpt;
+ }
+
+ protected void runAnyPendingCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight compaction at instant " + instant);
+ compact(instant.getTimestamp(), true);
+ });
+ }
+
+ protected void runAnyPendingLogCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingLogCompactionTimeline().getInstantsAsStream()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight log compaction at instant " + instant);
+ logCompact(instant.getTimestamp(), true);
+ });
+ }
+
+ /***
+ * Schedules compaction inline.
+ * @param extraMetadata extrametada to be used.
+ * @return compaction instant if scheduled.
+ */
+ protected Option<String> inlineScheduleCompaction(Option<Map<String, String>> extraMetadata) {
+ return scheduleCompaction(extraMetadata);
+ }
+
+ /**
+ * Schedules a new compaction instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected abstract HoodieWriteMetadata<O> compact(String compactionInstantTime, boolean shouldComplete);
+
+ /**
+ * Commit a compaction operation. Allow passing additional meta-data to be stored in commit instant file.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @param metadata All the metadata that gets stored along with a commit
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public abstract void commitCompaction(String compactionInstantTime, HoodieCommitMetadata metadata,
+ Option<Map<String, String>> extraMetadata);
+
+ /**
+ * Commit Compaction and track metrics.
+ */
+ protected abstract void completeCompaction(HoodieCommitMetadata metadata, HoodieTable table, String compactionCommitTime);
+
+ /**
+ * Schedules a new log compaction instant.
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleLogCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleLogCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new log compaction instant with passed-in instant time.
+ * @param instantTime Log Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleLogCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.LOG_COMPACT).isPresent();
+ }
+
+ /**
+ * Performs Log Compaction for the workload stored in instant-time.
+ *
+ * @param logCompactionInstantTime Log Compaction Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> logCompact(String logCompactionInstantTime) {
+ return logCompact(logCompactionInstantTime, config.shouldAutoCommit());
+ }
+
+ /**
+ * Commit Log Compaction and track metrics.
+ */
+ protected void completeLogCompaction(HoodieCommitMetadata metadata, HoodieTable table, String logCompactionCommitTime) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+
+ /**
+ * Schedules a new compaction instant with passed-in instant time.
+ *
+ * @param instantTime Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.COMPACT).isPresent();
+ }
+
+ /**
+ * Schedules a new clustering instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleClustering(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleClusteringAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new clustering instant with passed-in instant time.
+ *
+ * @param instantTime clustering Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleClusteringAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLUSTER).isPresent();
+ }
+
+ /**
+ * Schedules a new cleaning instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected Option<String> scheduleCleaning(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCleaningAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new cleaning instant with passed-in instant time.
+ *
+ * @param instantTime cleaning Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected boolean scheduleCleaningAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLEAN).isPresent();
+ }
+
+ /**
+ * Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.
+ *
+ * @param clusteringInstant Clustering Instant Time
+ * @return Collection of Write Status
+ */
+ public abstract HoodieWriteMetadata<O> cluster(String clusteringInstant, boolean shouldComplete);
+
+ protected void runTableServicesInline(HoodieTable table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
+ if (!tableServicesEnabled(config)) {
+ return;
+ }
+
+ if (!config.areAnyTableServicesExecutedInline() && !config.areAnyTableServicesScheduledInline()) {
+ return;
+ }
+
+ if (config.isMetadataTableEnabled()) {
+ table.getHoodieView().sync();
+ }
+ // Do an inline compaction if enabled
+ if (config.inlineCompactionEnabled()) {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
+ inlineCompaction(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().empty()) {
+ // proceed only if there are no pending compactions
+ metadata.addMetadata(HoodieCompactionConfig.SCHEDULE_INLINE_COMPACT.key(), "true");
+ inlineScheduleCompaction(extraMetadata);
+ }
+
+ // Do an inline log compaction if enabled
+ if (config.inlineLogCompactionEnabled()) {
+ runAnyPendingLogCompactions(table);
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "true");
+ inlineLogCompact(extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "false");
+ }
+
+ // Do an inline clustering if enabled
+ if (config.inlineClusteringEnabled()) {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "true");
+ inlineClustering(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineClusteringEnabled() && config.scheduleInlineClustering()
+ && table.getActiveTimeline().filterPendingReplaceTimeline().empty()) {
+ // proceed only if there are no pending clustering
+ metadata.addMetadata(HoodieClusteringConfig.SCHEDULE_INLINE_CLUSTERING.key(), "true");
+ inlineScheduleClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedule table services such as clustering, compaction & cleaning.
+ *
+ * @param extraMetadata Metadata to pass onto the scheduled service instant
+ * @param tableServiceType Type of table service to schedule
+ * @return
+ */
+ public Option<String> scheduleTableService(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ // A lock is required to guard against race conditions between an on-going writer and scheduling a table service.
+ final Option<HoodieInstant> inflightInstant = Option.of(new HoodieInstant(HoodieInstant.State.REQUESTED,
+ tableServiceType.getAction(), instantTime));
+ try {
+ this.txnManager.beginTransaction(inflightInstant, Option.empty());
+ LOG.info("Scheduling table service " + tableServiceType);
+ return scheduleTableServiceInternal(instantTime, extraMetadata, tableServiceType);
+ } finally {
+ this.txnManager.endTransaction(inflightInstant);
+ }
+ }
+
+ protected Option<String> scheduleTableServiceInternal(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ if (!tableServicesEnabled(config)) {
+ return Option.empty();
+ }
+
+ Option<String> option = Option.empty();
+ HoodieTable<?, ?, ?, ?> table = createTable(config, hadoopConf);
+
+ switch (tableServiceType) {
+ case ARCHIVE:
+ LOG.info("Scheduling archiving is not supported. Skipping.");
+ break;
+ case CLUSTER:
+ LOG.info("Scheduling clustering at instant time :" + instantTime);
+ Option<HoodieClusteringPlan> clusteringPlan = table
+ .scheduleClustering(context, instantTime, extraMetadata);
+ option = clusteringPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case COMPACT:
+ LOG.info("Scheduling compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> compactionPlan = table
+ .scheduleCompaction(context, instantTime, extraMetadata);
+ option = compactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case LOG_COMPACT:
+ LOG.info("Scheduling log compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> logCompactionPlan = table
+ .scheduleLogCompaction(context, instantTime, extraMetadata);
+ option = logCompactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case CLEAN:
+ LOG.info("Scheduling cleaning at instant time :" + instantTime);
+ Option<HoodieCleanerPlan> cleanerPlan = table
+ .scheduleCleaning(context, instantTime, extraMetadata);
+ option = cleanerPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ default:
+ throw new IllegalArgumentException("Invalid TableService " + tableServiceType);
+ }
+
+ Option<String> instantRange = delegateToTableServiceManager(tableServiceType, table);
+ if (instantRange.isPresent()) {
+ LOG.info("Delegate instant [" + instantRange.get() + "] to table service manager");
+ }
+
+ return option;
+ }
+
+ protected abstract HoodieTable<?, ?, ?, ?> createTable(HoodieWriteConfig config, Configuration hadoopConf);
+
+ /**
+ * Executes a clustering plan on a table, serially before or after an insert/upsert action.
+ * Schedules and executes clustering inline.
+ */
+ protected Option<String> inlineClustering(Option<Map<String, String>> extraMetadata) {
+ Option<String> clusteringInstantOpt = inlineScheduleClustering(extraMetadata);
+ clusteringInstantOpt.ifPresent(clusteringInstant -> {
+ // inline cluster should auto commit as the user is never given control
+ cluster(clusteringInstant, true);
+ });
+ return clusteringInstantOpt;
+ }
+
+ private void inlineClustering(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.replacecommit)) {
+ scheduleClustering(extraMetadata);
+ } else {
+ runAnyPendingClustering(table);
+ inlineClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedules clustering inline.
+ *
+ * @param extraMetadata extrametadata to use.
+ * @return clustering instant if scheduled.
+ */
+ protected Option<String> inlineScheduleClustering(Option<Map<String, String>> extraMetadata) {
+ return scheduleClustering(extraMetadata);
+ }
+
+ protected void runAnyPendingClustering(HoodieTable table) {
+ table.getActiveTimeline().filterPendingReplaceTimeline().getInstants().forEach(instant -> {
+ Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(table.getMetaClient(), instant);
+ if (instantPlan.isPresent()) {
+ LOG.info("Running pending clustering at instant " + instantPlan.get().getLeft());
+ cluster(instant.getTimestamp(), true);
+ }
+ });
+ }
+
+ /**
+ * Finalize Write operation.
+ *
+ * @param table HoodieTable
+ * @param instantTime Instant Time
+ * @param stats Hoodie Write Stat
+ */
+ protected void finalizeWrite(HoodieTable table, String instantTime, List<HoodieWriteStat> stats) {
Review Comment:
@yuzhaojing should this be extracted to base client?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -0,0 +1,840 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class BaseHoodieTableServiceClient<O> extends BaseHoodieClient implements RunsTableService {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+
+ protected BaseHoodieTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected void startAsyncCleanerService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncCleanerService == null) {
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(writeClient);
+ } else {
+ this.asyncCleanerService.start(null);
+ }
+ }
+
+ protected void startAsyncArchiveService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncArchiveService == null) {
+ this.asyncArchiveService = AsyncArchiveService.startAsyncArchiveIfEnabled(writeClient);
+ } else {
+ this.asyncArchiveService.start(null);
+ }
+ }
+
+ protected void asyncClean() {
+ AsyncCleanerService.waitForCompletion(asyncCleanerService);
+ }
+
+ protected void asyncArchive() {
+ AsyncArchiveService.waitForCompletion(asyncArchiveService);
+ }
+
+ protected void setTableServiceTimer(WriteOperationType operationType) {
+ switch (operationType) {
+ case CLUSTER:
+ clusteringTimer = metrics.getClusteringCtx();
+ break;
+ case COMPACT:
+ compactionTimer = metrics.getCompactionCtx();
+ break;
+ case LOG_COMPACT:
+ logCompactionTimer = metrics.getLogCompactionCtx();
+ break;
+ default:
+ }
+ }
+
+ /**
+ * Performs a compaction operation on a table, serially before or after an insert/upsert action.
+ * Scheduling and execution is done inline.
+ */
+ protected Option<String> inlineCompaction(Option<Map<String, String>> extraMetadata) {
+ Option<String> compactionInstantTimeOpt = inlineScheduleCompaction(extraMetadata);
+ compactionInstantTimeOpt.ifPresent(compactInstantTime -> {
+ // inline compaction should auto commit as the user is never given control
+ compact(compactInstantTime, true);
+ });
+ return compactionInstantTimeOpt;
+ }
+
+ private void inlineCompaction(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.compaction)) {
+ scheduleCompaction(extraMetadata);
+ } else {
+ runAnyPendingCompactions(table);
+ inlineCompaction(extraMetadata);
+ }
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Log Compaction for the workload stored in instant-time.s
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected HoodieWriteMetadata<O> logCompact(String compactionInstantTime, boolean shouldComplete) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+ /**
+ * Performs a log compaction operation on a table, serially before or after an insert/upsert action.
+ */
+ protected Option<String> inlineLogCompact(Option<Map<String, String>> extraMetadata) {
+ Option<String> logCompactionInstantTimeOpt = scheduleLogCompaction(extraMetadata);
+ logCompactionInstantTimeOpt.ifPresent(logCompactInstantTime -> {
+ // inline log compaction should auto commit as the user is never given control
+ logCompact(logCompactInstantTime, true);
+ });
+ return logCompactionInstantTimeOpt;
+ }
+
+ protected void runAnyPendingCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight compaction at instant " + instant);
+ compact(instant.getTimestamp(), true);
+ });
+ }
+
+ protected void runAnyPendingLogCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingLogCompactionTimeline().getInstantsAsStream()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight log compaction at instant " + instant);
+ logCompact(instant.getTimestamp(), true);
+ });
+ }
+
+ /***
+ * Schedules compaction inline.
+ * @param extraMetadata extrametada to be used.
+ * @return compaction instant if scheduled.
+ */
+ protected Option<String> inlineScheduleCompaction(Option<Map<String, String>> extraMetadata) {
+ return scheduleCompaction(extraMetadata);
+ }
+
+ /**
+ * Schedules a new compaction instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected abstract HoodieWriteMetadata<O> compact(String compactionInstantTime, boolean shouldComplete);
+
+ /**
+ * Commit a compaction operation. Allow passing additional meta-data to be stored in commit instant file.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @param metadata All the metadata that gets stored along with a commit
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public abstract void commitCompaction(String compactionInstantTime, HoodieCommitMetadata metadata,
+ Option<Map<String, String>> extraMetadata);
+
+ /**
+ * Commit Compaction and track metrics.
+ */
+ protected abstract void completeCompaction(HoodieCommitMetadata metadata, HoodieTable table, String compactionCommitTime);
+
+ /**
+ * Schedules a new log compaction instant.
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleLogCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleLogCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new log compaction instant with passed-in instant time.
+ * @param instantTime Log Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleLogCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.LOG_COMPACT).isPresent();
+ }
+
+ /**
+ * Performs Log Compaction for the workload stored in instant-time.
+ *
+ * @param logCompactionInstantTime Log Compaction Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> logCompact(String logCompactionInstantTime) {
+ return logCompact(logCompactionInstantTime, config.shouldAutoCommit());
+ }
+
+ /**
+ * Commit Log Compaction and track metrics.
+ */
+ protected void completeLogCompaction(HoodieCommitMetadata metadata, HoodieTable table, String logCompactionCommitTime) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+
+ /**
+ * Schedules a new compaction instant with passed-in instant time.
+ *
+ * @param instantTime Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.COMPACT).isPresent();
+ }
+
+ /**
+ * Schedules a new clustering instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleClustering(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleClusteringAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new clustering instant with passed-in instant time.
+ *
+ * @param instantTime clustering Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleClusteringAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLUSTER).isPresent();
+ }
+
+ /**
+ * Schedules a new cleaning instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected Option<String> scheduleCleaning(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCleaningAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new cleaning instant with passed-in instant time.
+ *
+ * @param instantTime cleaning Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected boolean scheduleCleaningAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLEAN).isPresent();
+ }
+
+ /**
+ * Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.
+ *
+ * @param clusteringInstant Clustering Instant Time
+ * @return Collection of Write Status
+ */
+ public abstract HoodieWriteMetadata<O> cluster(String clusteringInstant, boolean shouldComplete);
+
+ protected void runTableServicesInline(HoodieTable table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
+ if (!tableServicesEnabled(config)) {
+ return;
+ }
+
+ if (!config.areAnyTableServicesExecutedInline() && !config.areAnyTableServicesScheduledInline()) {
+ return;
+ }
+
+ if (config.isMetadataTableEnabled()) {
+ table.getHoodieView().sync();
+ }
+ // Do an inline compaction if enabled
+ if (config.inlineCompactionEnabled()) {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
+ inlineCompaction(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().empty()) {
+ // proceed only if there are no pending compactions
+ metadata.addMetadata(HoodieCompactionConfig.SCHEDULE_INLINE_COMPACT.key(), "true");
+ inlineScheduleCompaction(extraMetadata);
+ }
+
+ // Do an inline log compaction if enabled
+ if (config.inlineLogCompactionEnabled()) {
+ runAnyPendingLogCompactions(table);
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "true");
+ inlineLogCompact(extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "false");
+ }
+
+ // Do an inline clustering if enabled
+ if (config.inlineClusteringEnabled()) {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "true");
+ inlineClustering(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineClusteringEnabled() && config.scheduleInlineClustering()
+ && table.getActiveTimeline().filterPendingReplaceTimeline().empty()) {
+ // proceed only if there are no pending clustering
+ metadata.addMetadata(HoodieClusteringConfig.SCHEDULE_INLINE_CLUSTERING.key(), "true");
+ inlineScheduleClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedule table services such as clustering, compaction & cleaning.
+ *
+ * @param extraMetadata Metadata to pass onto the scheduled service instant
+ * @param tableServiceType Type of table service to schedule
+ * @return
+ */
+ public Option<String> scheduleTableService(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ // A lock is required to guard against race conditions between an on-going writer and scheduling a table service.
+ final Option<HoodieInstant> inflightInstant = Option.of(new HoodieInstant(HoodieInstant.State.REQUESTED,
+ tableServiceType.getAction(), instantTime));
+ try {
+ this.txnManager.beginTransaction(inflightInstant, Option.empty());
+ LOG.info("Scheduling table service " + tableServiceType);
+ return scheduleTableServiceInternal(instantTime, extraMetadata, tableServiceType);
+ } finally {
+ this.txnManager.endTransaction(inflightInstant);
+ }
+ }
+
+ protected Option<String> scheduleTableServiceInternal(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ if (!tableServicesEnabled(config)) {
+ return Option.empty();
+ }
+
+ Option<String> option = Option.empty();
+ HoodieTable<?, ?, ?, ?> table = createTable(config, hadoopConf);
+
+ switch (tableServiceType) {
+ case ARCHIVE:
+ LOG.info("Scheduling archiving is not supported. Skipping.");
+ break;
+ case CLUSTER:
+ LOG.info("Scheduling clustering at instant time :" + instantTime);
+ Option<HoodieClusteringPlan> clusteringPlan = table
+ .scheduleClustering(context, instantTime, extraMetadata);
+ option = clusteringPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case COMPACT:
+ LOG.info("Scheduling compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> compactionPlan = table
+ .scheduleCompaction(context, instantTime, extraMetadata);
+ option = compactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case LOG_COMPACT:
+ LOG.info("Scheduling log compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> logCompactionPlan = table
+ .scheduleLogCompaction(context, instantTime, extraMetadata);
+ option = logCompactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case CLEAN:
+ LOG.info("Scheduling cleaning at instant time :" + instantTime);
+ Option<HoodieCleanerPlan> cleanerPlan = table
+ .scheduleCleaning(context, instantTime, extraMetadata);
+ option = cleanerPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ default:
+ throw new IllegalArgumentException("Invalid TableService " + tableServiceType);
+ }
+
+ Option<String> instantRange = delegateToTableServiceManager(tableServiceType, table);
+ if (instantRange.isPresent()) {
+ LOG.info("Delegate instant [" + instantRange.get() + "] to table service manager");
+ }
+
+ return option;
+ }
+
+ protected abstract HoodieTable<?, ?, ?, ?> createTable(HoodieWriteConfig config, Configuration hadoopConf);
+
+ /**
+ * Executes a clustering plan on a table, serially before or after an insert/upsert action.
+ * Schedules and executes clustering inline.
+ */
+ protected Option<String> inlineClustering(Option<Map<String, String>> extraMetadata) {
+ Option<String> clusteringInstantOpt = inlineScheduleClustering(extraMetadata);
+ clusteringInstantOpt.ifPresent(clusteringInstant -> {
+ // inline cluster should auto commit as the user is never given control
+ cluster(clusteringInstant, true);
+ });
+ return clusteringInstantOpt;
+ }
+
+ private void inlineClustering(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.replacecommit)) {
+ scheduleClustering(extraMetadata);
+ } else {
+ runAnyPendingClustering(table);
+ inlineClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedules clustering inline.
+ *
+ * @param extraMetadata extrametadata to use.
+ * @return clustering instant if scheduled.
+ */
+ protected Option<String> inlineScheduleClustering(Option<Map<String, String>> extraMetadata) {
+ return scheduleClustering(extraMetadata);
+ }
+
+ protected void runAnyPendingClustering(HoodieTable table) {
+ table.getActiveTimeline().filterPendingReplaceTimeline().getInstants().forEach(instant -> {
+ Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(table.getMetaClient(), instant);
+ if (instantPlan.isPresent()) {
+ LOG.info("Running pending clustering at instant " + instantPlan.get().getLeft());
+ cluster(instant.getTimestamp(), true);
+ }
+ });
+ }
+
+ /**
+ * Finalize Write operation.
+ *
+ * @param table HoodieTable
+ * @param instantTime Instant Time
+ * @param stats Hoodie Write Stat
+ */
+ protected void finalizeWrite(HoodieTable table, String instantTime, List<HoodieWriteStat> stats) {
+ try {
+ final Timer.Context finalizeCtx = metrics.getFinalizeCtx();
+ table.finalizeWrite(context, instantTime, stats);
+ if (finalizeCtx != null) {
+ Option<Long> durationInMs = Option.of(metrics.getDurationInMs(finalizeCtx.stop()));
+ durationInMs.ifPresent(duration -> {
+ LOG.info("Finalize write elapsed time (milliseconds): " + duration);
+ metrics.updateFinalizeWriteMetrics(duration, stats.size());
+ });
+ }
+ } catch (HoodieIOException ioe) {
+ throw new HoodieCommitException("Failed to complete commit " + instantTime + " due to finalize errors.", ioe);
+ }
+ }
+
+ /**
+ * Write the HoodieCommitMetadata to metadata table if available.
+ *
+ * @param table {@link HoodieTable} of interest.
+ * @param instantTime instant time of the commit.
+ * @param actionType action type of the commit.
+ * @param metadata instance of {@link HoodieCommitMetadata}.
+ */
+ protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
+ context.setJobStatus(this.getClass().getSimpleName(), "Committing to metadata table: " + config.getTableName());
+ table.getMetadataWriter(instantTime).ifPresent(w -> ((HoodieTableMetadataWriter) w).update(metadata, instantTime,
+ table.isTableServiceAction(actionType, instantTime)));
Review Comment:
`writeTableMetadata()` in the base write client should delegate to table service client when the action is table service. Though the implementation is the equivalent now, from API standpoint, we should make it delegated accordingly
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -352,182 +303,23 @@ public void commitLogCompaction(String logCompactionInstantTime, HoodieCommitMet
protected void completeLogCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String logCompactionCommitTime) {
- this.context.setJobStatus(this.getClass().getSimpleName(), "Collect log compaction write status and commit compaction");
- List<HoodieWriteStat> writeStats = metadata.getWriteStats();
final HoodieInstant logCompactionInstant = new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.LOG_COMPACTION_ACTION, logCompactionCommitTime);
- try {
- this.txnManager.beginTransaction(Option.of(logCompactionInstant), Option.empty());
- preCommit(logCompactionInstant, metadata);
- finalizeWrite(table, logCompactionCommitTime, writeStats);
- // commit to data table after committing to metadata table.
- updateTableMetadata(table, metadata, logCompactionInstant);
- LOG.info("Committing Log Compaction " + logCompactionCommitTime + ". Finished with result " + metadata);
- CompactHelpers.getInstance().completeInflightLogCompaction(table, logCompactionCommitTime, metadata);
- } finally {
- this.txnManager.endTransaction(Option.of(logCompactionInstant));
- }
- WriteMarkersFactory.get(config.getMarkersType(), table, logCompactionCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
- if (compactionTimer != null) {
- long durationInMs = metrics.getDurationInMs(compactionTimer.stop());
- HoodieActiveTimeline.parseDateFromInstantTimeSafely(logCompactionCommitTime).ifPresent(parsedInstant ->
- metrics.updateCommitMetrics(parsedInstant.getTime(), durationInMs, metadata, HoodieActiveTimeline.LOG_COMPACTION_ACTION)
- );
- }
- LOG.info("Log Compacted successfully on commit " + logCompactionCommitTime);
+ preCommit(logCompactionInstant, metadata);
+ tableServiceClient.completeLogCompaction(metadata, table, logCompactionCommitTime);
}
@Override
protected HoodieWriteMetadata<JavaRDD<WriteStatus>> logCompact(String logCompactionInstantTime, boolean shouldComplete) {
HoodieSparkTable<T> table = HoodieSparkTable.create(config, context);
preWrite(logCompactionInstantTime, WriteOperationType.LOG_COMPACT, table.getMetaClient());
- HoodieTimeline pendingLogCompactionTimeline = table.getActiveTimeline().filterPendingLogCompactionTimeline();
- HoodieInstant inflightInstant = HoodieTimeline.getLogCompactionInflightInstant(logCompactionInstantTime);
- if (pendingLogCompactionTimeline.containsInstant(inflightInstant)) {
- LOG.info("Found Log compaction inflight file. Rolling back the commit and exiting.");
- table.rollbackInflightLogCompaction(inflightInstant, commitToRollback -> getPendingRollbackInfo(table.getMetaClient(), commitToRollback, false));
- table.getMetaClient().reloadActiveTimeline();
- throw new HoodieException("Inflight logcompaction file exists");
- }
- logCompactionTimer = metrics.getLogCompactionCtx();
- WriteMarkersFactory.get(config.getMarkersType(), table, logCompactionInstantTime);
- HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata = table.logCompact(context, logCompactionInstantTime);
- HoodieWriteMetadata<JavaRDD<WriteStatus>> logCompactionMetadata = writeMetadata.clone(HoodieJavaRDD.getJavaRDD(writeMetadata.getWriteStatuses()));
- if (shouldComplete && logCompactionMetadata.getCommitMetadata().isPresent()) {
- completeTableService(TableServiceType.LOG_COMPACT, logCompactionMetadata.getCommitMetadata().get(), table, logCompactionInstantTime);
- }
- return logCompactionMetadata;
+ return tableServiceClient.logCompact(logCompactionInstantTime, shouldComplete);
Review Comment:
if you force cast `tableServiceClient` here, it'll be easier to navigate through IDE
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -283,44 +270,14 @@ public void preWrite(String instantTime, WriteOperationType writeOperationType,
@Override
protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
- try (HoodieBackedTableMetadataWriter metadataWriter = initMetadataWriter()) {
- metadataWriter.update(metadata, instantTime, getHoodieTable().isTableServiceAction(actionType, instantTime));
- } catch (Exception e) {
- throw new HoodieException("Failed to update metadata", e);
- }
- }
-
- /**
- * Initialize the table metadata writer, for e.g, bootstrap the metadata table
- * from the filesystem if it does not exist.
- */
- private HoodieBackedTableMetadataWriter initMetadataWriter() {
- return (HoodieBackedTableMetadataWriter) FlinkHoodieBackedTableMetadataWriter.create(
- FlinkClientUtil.getHadoopConf(), this.config, HoodieFlinkEngineContext.DEFAULT);
+ tableServiceClient.writeTableMetadata(table, instantTime, actionType, metadata);
Review Comment:
flink write client always delegates this to table service client, which differs from the base write client behavior. Similarly let's follow up a task to consolidate this.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -352,182 +303,23 @@ public void commitLogCompaction(String logCompactionInstantTime, HoodieCommitMet
protected void completeLogCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String logCompactionCommitTime) {
- this.context.setJobStatus(this.getClass().getSimpleName(), "Collect log compaction write status and commit compaction");
- List<HoodieWriteStat> writeStats = metadata.getWriteStats();
final HoodieInstant logCompactionInstant = new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.LOG_COMPACTION_ACTION, logCompactionCommitTime);
- try {
- this.txnManager.beginTransaction(Option.of(logCompactionInstant), Option.empty());
- preCommit(logCompactionInstant, metadata);
- finalizeWrite(table, logCompactionCommitTime, writeStats);
- // commit to data table after committing to metadata table.
- updateTableMetadata(table, metadata, logCompactionInstant);
- LOG.info("Committing Log Compaction " + logCompactionCommitTime + ". Finished with result " + metadata);
- CompactHelpers.getInstance().completeInflightLogCompaction(table, logCompactionCommitTime, metadata);
- } finally {
- this.txnManager.endTransaction(Option.of(logCompactionInstant));
- }
- WriteMarkersFactory.get(config.getMarkersType(), table, logCompactionCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
- if (compactionTimer != null) {
- long durationInMs = metrics.getDurationInMs(compactionTimer.stop());
- HoodieActiveTimeline.parseDateFromInstantTimeSafely(logCompactionCommitTime).ifPresent(parsedInstant ->
- metrics.updateCommitMetrics(parsedInstant.getTime(), durationInMs, metadata, HoodieActiveTimeline.LOG_COMPACTION_ACTION)
- );
- }
- LOG.info("Log Compacted successfully on commit " + logCompactionCommitTime);
+ preCommit(logCompactionInstant, metadata);
+ tableServiceClient.completeLogCompaction(metadata, table, logCompactionCommitTime);
}
@Override
protected HoodieWriteMetadata<JavaRDD<WriteStatus>> logCompact(String logCompactionInstantTime, boolean shouldComplete) {
HoodieSparkTable<T> table = HoodieSparkTable.create(config, context);
preWrite(logCompactionInstantTime, WriteOperationType.LOG_COMPACT, table.getMetaClient());
- HoodieTimeline pendingLogCompactionTimeline = table.getActiveTimeline().filterPendingLogCompactionTimeline();
- HoodieInstant inflightInstant = HoodieTimeline.getLogCompactionInflightInstant(logCompactionInstantTime);
- if (pendingLogCompactionTimeline.containsInstant(inflightInstant)) {
- LOG.info("Found Log compaction inflight file. Rolling back the commit and exiting.");
- table.rollbackInflightLogCompaction(inflightInstant, commitToRollback -> getPendingRollbackInfo(table.getMetaClient(), commitToRollback, false));
- table.getMetaClient().reloadActiveTimeline();
- throw new HoodieException("Inflight logcompaction file exists");
- }
- logCompactionTimer = metrics.getLogCompactionCtx();
- WriteMarkersFactory.get(config.getMarkersType(), table, logCompactionInstantTime);
- HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata = table.logCompact(context, logCompactionInstantTime);
- HoodieWriteMetadata<JavaRDD<WriteStatus>> logCompactionMetadata = writeMetadata.clone(HoodieJavaRDD.getJavaRDD(writeMetadata.getWriteStatuses()));
- if (shouldComplete && logCompactionMetadata.getCommitMetadata().isPresent()) {
- completeTableService(TableServiceType.LOG_COMPACT, logCompactionMetadata.getCommitMetadata().get(), table, logCompactionInstantTime);
- }
- return logCompactionMetadata;
+ return tableServiceClient.logCompact(logCompactionInstantTime, shouldComplete);
}
@Override
public HoodieWriteMetadata<JavaRDD<WriteStatus>> cluster(String clusteringInstant, boolean shouldComplete) {
HoodieSparkTable<T> table = HoodieSparkTable.create(config, context);
preWrite(clusteringInstant, WriteOperationType.CLUSTER, table.getMetaClient());
- HoodieTimeline pendingClusteringTimeline = table.getActiveTimeline().filterPendingReplaceTimeline();
- HoodieInstant inflightInstant = HoodieTimeline.getReplaceCommitInflightInstant(clusteringInstant);
- if (pendingClusteringTimeline.containsInstant(inflightInstant)) {
- table.rollbackInflightClustering(inflightInstant, commitToRollback -> getPendingRollbackInfo(table.getMetaClient(), commitToRollback, false));
- table.getMetaClient().reloadActiveTimeline();
- }
- clusteringTimer = metrics.getClusteringCtx();
- LOG.info("Starting clustering at " + clusteringInstant);
- HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata = table.cluster(context, clusteringInstant);
- HoodieWriteMetadata<JavaRDD<WriteStatus>> clusteringMetadata = writeMetadata.clone(HoodieJavaRDD.getJavaRDD(writeMetadata.getWriteStatuses()));
- // Validation has to be done after cloning. if not, it could result in dereferencing the write status twice which means clustering could get executed twice.
- validateClusteringCommit(clusteringMetadata, clusteringInstant, table);
- // TODO : Where is shouldComplete used ?
- if (shouldComplete && clusteringMetadata.getCommitMetadata().isPresent()) {
- completeTableService(TableServiceType.CLUSTER, clusteringMetadata.getCommitMetadata().get(), table, clusteringInstant);
- }
- return clusteringMetadata;
- }
-
- private void completeClustering(HoodieReplaceCommitMetadata metadata,
- HoodieTable table,
- String clusteringCommitTime) {
- List<HoodieWriteStat> writeStats = metadata.getPartitionToWriteStats().entrySet().stream().flatMap(e ->
- e.getValue().stream()).collect(Collectors.toList());
-
- if (writeStats.stream().mapToLong(HoodieWriteStat::getTotalWriteErrors).sum() > 0) {
- throw new HoodieClusteringException("Clustering failed to write to files:"
- + writeStats.stream().filter(s -> s.getTotalWriteErrors() > 0L).map(HoodieWriteStat::getFileId).collect(Collectors.joining(",")));
- }
-
- final HoodieInstant clusteringInstant = HoodieTimeline.getReplaceCommitInflightInstant(clusteringCommitTime);
- try {
- this.txnManager.beginTransaction(Option.of(clusteringInstant), Option.empty());
-
- finalizeWrite(table, clusteringCommitTime, writeStats);
- // Update table's metadata (table)
- updateTableMetadata(table, metadata, clusteringInstant);
-
- LOG.info("Committing Clustering " + clusteringCommitTime + ". Finished with result " + metadata);
-
- table.getActiveTimeline().transitionReplaceInflightToComplete(
- clusteringInstant,
- Option.of(metadata.toJsonString().getBytes(StandardCharsets.UTF_8)));
- } catch (Exception e) {
- throw new HoodieClusteringException("unable to transition clustering inflight to complete: " + clusteringCommitTime, e);
- } finally {
- this.txnManager.endTransaction(Option.of(clusteringInstant));
- }
- WriteMarkersFactory.get(config.getMarkersType(), table, clusteringCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
- if (clusteringTimer != null) {
- long durationInMs = metrics.getDurationInMs(clusteringTimer.stop());
- HoodieActiveTimeline.parseDateFromInstantTimeSafely(clusteringCommitTime).ifPresent(parsedInstant ->
- metrics.updateCommitMetrics(parsedInstant.getTime(), durationInMs, metadata, HoodieActiveTimeline.REPLACE_COMMIT_ACTION)
- );
- }
- LOG.info("Clustering successfully on commit " + clusteringCommitTime);
- }
-
- private void validateClusteringCommit(HoodieWriteMetadata<JavaRDD<WriteStatus>> clusteringMetadata, String clusteringCommitTime, HoodieTable table) {
- if (clusteringMetadata.getWriteStatuses().isEmpty()) {
- HoodieClusteringPlan clusteringPlan = ClusteringUtils.getClusteringPlan(
- table.getMetaClient(), HoodieTimeline.getReplaceCommitRequestedInstant(clusteringCommitTime))
- .map(Pair::getRight).orElseThrow(() -> new HoodieClusteringException(
- "Unable to read clustering plan for instant: " + clusteringCommitTime));
- throw new HoodieClusteringException("Clustering plan produced 0 WriteStatus for " + clusteringCommitTime
- + " #groups: " + clusteringPlan.getInputGroups().size() + " expected at least "
- + clusteringPlan.getInputGroups().stream().mapToInt(HoodieClusteringGroup::getNumOutputFileGroups).sum()
- + " write statuses");
- }
- }
-
- private void updateTableMetadata(HoodieTable table, HoodieCommitMetadata commitMetadata,
- HoodieInstant hoodieInstant) {
- boolean isTableServiceAction = table.isTableServiceAction(hoodieInstant.getAction(), hoodieInstant.getTimestamp());
- // Do not do any conflict resolution here as we do with regular writes. We take the lock here to ensure all writes to metadata table happens within a
- // single lock (single writer). Because more than one write to metadata table will result in conflicts since all of them updates the same partition.
- table.getMetadataWriter(hoodieInstant.getTimestamp())
- .ifPresent(writer -> ((HoodieTableMetadataWriter) writer).update(commitMetadata, hoodieInstant.getTimestamp(), isTableServiceAction));
- }
-
- @Override
- protected void initMetadataTable(Option<String> instantTime) {
- // Initialize Metadata Table to make sure it's bootstrapped _before_ the operation,
- // if it didn't exist before
- // See https://issues.apache.org/jira/browse/HUDI-3343 for more details
- initializeMetadataTable(instantTime);
- }
-
- /**
- * Initialize the metadata table if needed. Creating the metadata table writer
- * will trigger the initial bootstrapping from the data table.
- *
- * @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
- */
- private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
- }
- }
- }
Review Comment:
these 2 not used by spark table service client? again, by design they fit better with write client not table service client
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java:
##########
@@ -283,44 +270,14 @@ public void preWrite(String instantTime, WriteOperationType writeOperationType,
@Override
protected void writeTableMetadata(HoodieTable table, String instantTime, String actionType, HoodieCommitMetadata metadata) {
- try (HoodieBackedTableMetadataWriter metadataWriter = initMetadataWriter()) {
- metadataWriter.update(metadata, instantTime, getHoodieTable().isTableServiceAction(actionType, instantTime));
- } catch (Exception e) {
- throw new HoodieException("Failed to update metadata", e);
- }
- }
-
- /**
- * Initialize the table metadata writer, for e.g, bootstrap the metadata table
- * from the filesystem if it does not exist.
- */
- private HoodieBackedTableMetadataWriter initMetadataWriter() {
- return (HoodieBackedTableMetadataWriter) FlinkHoodieBackedTableMetadataWriter.create(
- FlinkClientUtil.getHadoopConf(), this.config, HoodieFlinkEngineContext.DEFAULT);
+ tableServiceClient.writeTableMetadata(table, instantTime, actionType, metadata);
}
/**
* Initialized the metadata table on start up, should only be called once on driver.
*/
public void initMetadataTable() {
- HoodieFlinkTable<?> table = getHoodieTable();
- if (config.isMetadataTableEnabled()) {
- // initialize the metadata table path
- // guard the metadata writer with concurrent lock
- try {
- this.txnManager.getLockManager().lock();
- initMetadataWriter().close();
- } catch (Exception e) {
- throw new HoodieException("Failed to initialize metadata table", e);
- } finally {
- this.txnManager.getLockManager().unlock();
- }
- // clean the obsolete index stats
- table.deleteMetadataIndexIfNecessary();
- } else {
- // delete the metadata table if it was enabled but is now disabled
- table.maybeDeleteMetadataTable();
- }
+ ((HoodieFlinkTableServiceClient<T>) tableServiceClient).initMetadataTable();
Review Comment:
by design initMetadataTable should be responsibility of a write client. Can you file a follow up task: consolidate the way MDT is initialized. i see that `org.apache.hudi.client.BaseHoodieWriteClient#doInitTable` and `org.apache.hudi.client.BaseHoodieWriteClient#initMetadataTable` are skipped by flink write client. we should standardize the clients' lifecycles.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -0,0 +1,840 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.async.AsyncArchiveService;
+import org.apache.hudi.async.AsyncCleanerService;
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.avro.model.HoodieCompactionPlan;
+import org.apache.hudi.avro.model.HoodieRollbackMetadata;
+import org.apache.hudi.avro.model.HoodieRollbackPlan;
+import org.apache.hudi.client.heartbeat.HeartbeatUtils;
+import org.apache.hudi.common.HoodiePendingRollbackInfo;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.ActionType;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.TableServiceType;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.CleanerUtils;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieCommitException;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieRollbackException;
+import org.apache.hudi.metadata.HoodieTableMetadataWriter;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.rollback.RollbackUtils;
+
+import com.codahale.metrics.Timer;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class BaseHoodieTableServiceClient<O> extends BaseHoodieClient implements RunsTableService {
+
+ private static final Logger LOG = LogManager.getLogger(BaseHoodieWriteClient.class);
+
+ protected transient Timer.Context compactionTimer;
+ protected transient Timer.Context clusteringTimer;
+ protected transient Timer.Context logCompactionTimer;
+
+ protected transient AsyncCleanerService asyncCleanerService;
+ protected transient AsyncArchiveService asyncArchiveService;
+
+ protected BaseHoodieTableServiceClient(HoodieEngineContext context, HoodieWriteConfig clientConfig) {
+ super(context, clientConfig, Option.empty());
+ }
+
+ protected void startAsyncCleanerService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncCleanerService == null) {
+ this.asyncCleanerService = AsyncCleanerService.startAsyncCleaningIfEnabled(writeClient);
+ } else {
+ this.asyncCleanerService.start(null);
+ }
+ }
+
+ protected void startAsyncArchiveService(BaseHoodieWriteClient writeClient) {
+ if (this.asyncArchiveService == null) {
+ this.asyncArchiveService = AsyncArchiveService.startAsyncArchiveIfEnabled(writeClient);
+ } else {
+ this.asyncArchiveService.start(null);
+ }
+ }
+
+ protected void asyncClean() {
+ AsyncCleanerService.waitForCompletion(asyncCleanerService);
+ }
+
+ protected void asyncArchive() {
+ AsyncArchiveService.waitForCompletion(asyncArchiveService);
+ }
+
+ protected void setTableServiceTimer(WriteOperationType operationType) {
+ switch (operationType) {
+ case CLUSTER:
+ clusteringTimer = metrics.getClusteringCtx();
+ break;
+ case COMPACT:
+ compactionTimer = metrics.getCompactionCtx();
+ break;
+ case LOG_COMPACT:
+ logCompactionTimer = metrics.getLogCompactionCtx();
+ break;
+ default:
+ }
+ }
+
+ /**
+ * Performs a compaction operation on a table, serially before or after an insert/upsert action.
+ * Scheduling and execution is done inline.
+ */
+ protected Option<String> inlineCompaction(Option<Map<String, String>> extraMetadata) {
+ Option<String> compactionInstantTimeOpt = inlineScheduleCompaction(extraMetadata);
+ compactionInstantTimeOpt.ifPresent(compactInstantTime -> {
+ // inline compaction should auto commit as the user is never given control
+ compact(compactInstantTime, true);
+ });
+ return compactionInstantTimeOpt;
+ }
+
+ private void inlineCompaction(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.compaction)) {
+ scheduleCompaction(extraMetadata);
+ } else {
+ runAnyPendingCompactions(table);
+ inlineCompaction(extraMetadata);
+ }
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Log Compaction for the workload stored in instant-time.s
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected HoodieWriteMetadata<O> logCompact(String compactionInstantTime, boolean shouldComplete) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+ /**
+ * Performs a log compaction operation on a table, serially before or after an insert/upsert action.
+ */
+ protected Option<String> inlineLogCompact(Option<Map<String, String>> extraMetadata) {
+ Option<String> logCompactionInstantTimeOpt = scheduleLogCompaction(extraMetadata);
+ logCompactionInstantTimeOpt.ifPresent(logCompactInstantTime -> {
+ // inline log compaction should auto commit as the user is never given control
+ logCompact(logCompactInstantTime, true);
+ });
+ return logCompactionInstantTimeOpt;
+ }
+
+ protected void runAnyPendingCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().getInstants()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight compaction at instant " + instant);
+ compact(instant.getTimestamp(), true);
+ });
+ }
+
+ protected void runAnyPendingLogCompactions(HoodieTable table) {
+ table.getActiveTimeline().getWriteTimeline().filterPendingLogCompactionTimeline().getInstantsAsStream()
+ .forEach(instant -> {
+ LOG.info("Running previously failed inflight log compaction at instant " + instant);
+ logCompact(instant.getTimestamp(), true);
+ });
+ }
+
+ /***
+ * Schedules compaction inline.
+ * @param extraMetadata extrametada to be used.
+ * @return compaction instant if scheduled.
+ */
+ protected Option<String> inlineScheduleCompaction(Option<Map<String, String>> extraMetadata) {
+ return scheduleCompaction(extraMetadata);
+ }
+
+ /**
+ * Schedules a new compaction instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Ensures compaction instant is in expected state and performs Compaction for the workload stored in instant-time.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @return Collection of Write Status
+ */
+ protected abstract HoodieWriteMetadata<O> compact(String compactionInstantTime, boolean shouldComplete);
+
+ /**
+ * Commit a compaction operation. Allow passing additional meta-data to be stored in commit instant file.
+ *
+ * @param compactionInstantTime Compaction Instant Time
+ * @param metadata All the metadata that gets stored along with a commit
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public abstract void commitCompaction(String compactionInstantTime, HoodieCommitMetadata metadata,
+ Option<Map<String, String>> extraMetadata);
+
+ /**
+ * Commit Compaction and track metrics.
+ */
+ protected abstract void completeCompaction(HoodieCommitMetadata metadata, HoodieTable table, String compactionCommitTime);
+
+ /**
+ * Schedules a new log compaction instant.
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleLogCompaction(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleLogCompactionAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new log compaction instant with passed-in instant time.
+ * @param instantTime Log Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleLogCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.LOG_COMPACT).isPresent();
+ }
+
+ /**
+ * Performs Log Compaction for the workload stored in instant-time.
+ *
+ * @param logCompactionInstantTime Log Compaction Instant Time
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public HoodieWriteMetadata<O> logCompact(String logCompactionInstantTime) {
+ return logCompact(logCompactionInstantTime, config.shouldAutoCommit());
+ }
+
+ /**
+ * Commit Log Compaction and track metrics.
+ */
+ protected void completeLogCompaction(HoodieCommitMetadata metadata, HoodieTable table, String logCompactionCommitTime) {
+ throw new UnsupportedOperationException("Log compaction is not supported yet.");
+ }
+
+
+ /**
+ * Schedules a new compaction instant with passed-in instant time.
+ *
+ * @param instantTime Compaction Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleCompactionAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.COMPACT).isPresent();
+ }
+
+ /**
+ * Schedules a new clustering instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public Option<String> scheduleClustering(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleClusteringAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new clustering instant with passed-in instant time.
+ *
+ * @param instantTime clustering Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ public boolean scheduleClusteringAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLUSTER).isPresent();
+ }
+
+ /**
+ * Schedules a new cleaning instant.
+ *
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected Option<String> scheduleCleaning(Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ return scheduleCleaningAtInstant(instantTime, extraMetadata) ? Option.of(instantTime) : Option.empty();
+ }
+
+ /**
+ * Schedules a new cleaning instant with passed-in instant time.
+ *
+ * @param instantTime cleaning Instant Time
+ * @param extraMetadata Extra Metadata to be stored
+ */
+ protected boolean scheduleCleaningAtInstant(String instantTime, Option<Map<String, String>> extraMetadata) throws HoodieIOException {
+ return scheduleTableService(instantTime, extraMetadata, TableServiceType.CLEAN).isPresent();
+ }
+
+ /**
+ * Ensures clustering instant is in expected state and performs clustering for the plan stored in metadata.
+ *
+ * @param clusteringInstant Clustering Instant Time
+ * @return Collection of Write Status
+ */
+ public abstract HoodieWriteMetadata<O> cluster(String clusteringInstant, boolean shouldComplete);
+
+ protected void runTableServicesInline(HoodieTable table, HoodieCommitMetadata metadata, Option<Map<String, String>> extraMetadata) {
+ if (!tableServicesEnabled(config)) {
+ return;
+ }
+
+ if (!config.areAnyTableServicesExecutedInline() && !config.areAnyTableServicesScheduledInline()) {
+ return;
+ }
+
+ if (config.isMetadataTableEnabled()) {
+ table.getHoodieView().sync();
+ }
+ // Do an inline compaction if enabled
+ if (config.inlineCompactionEnabled()) {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
+ inlineCompaction(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_COMPACT.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineCompactionEnabled() && config.scheduleInlineCompaction()
+ && table.getActiveTimeline().getWriteTimeline().filterPendingCompactionTimeline().empty()) {
+ // proceed only if there are no pending compactions
+ metadata.addMetadata(HoodieCompactionConfig.SCHEDULE_INLINE_COMPACT.key(), "true");
+ inlineScheduleCompaction(extraMetadata);
+ }
+
+ // Do an inline log compaction if enabled
+ if (config.inlineLogCompactionEnabled()) {
+ runAnyPendingLogCompactions(table);
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "true");
+ inlineLogCompact(extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieCompactionConfig.INLINE_LOG_COMPACT.key(), "false");
+ }
+
+ // Do an inline clustering if enabled
+ if (config.inlineClusteringEnabled()) {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "true");
+ inlineClustering(table, extraMetadata);
+ } else {
+ metadata.addMetadata(HoodieClusteringConfig.INLINE_CLUSTERING.key(), "false");
+ }
+
+ // if just inline schedule is enabled
+ if (!config.inlineClusteringEnabled() && config.scheduleInlineClustering()
+ && table.getActiveTimeline().filterPendingReplaceTimeline().empty()) {
+ // proceed only if there are no pending clustering
+ metadata.addMetadata(HoodieClusteringConfig.SCHEDULE_INLINE_CLUSTERING.key(), "true");
+ inlineScheduleClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedule table services such as clustering, compaction & cleaning.
+ *
+ * @param extraMetadata Metadata to pass onto the scheduled service instant
+ * @param tableServiceType Type of table service to schedule
+ * @return
+ */
+ public Option<String> scheduleTableService(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ // A lock is required to guard against race conditions between an on-going writer and scheduling a table service.
+ final Option<HoodieInstant> inflightInstant = Option.of(new HoodieInstant(HoodieInstant.State.REQUESTED,
+ tableServiceType.getAction(), instantTime));
+ try {
+ this.txnManager.beginTransaction(inflightInstant, Option.empty());
+ LOG.info("Scheduling table service " + tableServiceType);
+ return scheduleTableServiceInternal(instantTime, extraMetadata, tableServiceType);
+ } finally {
+ this.txnManager.endTransaction(inflightInstant);
+ }
+ }
+
+ protected Option<String> scheduleTableServiceInternal(String instantTime, Option<Map<String, String>> extraMetadata,
+ TableServiceType tableServiceType) {
+ if (!tableServicesEnabled(config)) {
+ return Option.empty();
+ }
+
+ Option<String> option = Option.empty();
+ HoodieTable<?, ?, ?, ?> table = createTable(config, hadoopConf);
+
+ switch (tableServiceType) {
+ case ARCHIVE:
+ LOG.info("Scheduling archiving is not supported. Skipping.");
+ break;
+ case CLUSTER:
+ LOG.info("Scheduling clustering at instant time :" + instantTime);
+ Option<HoodieClusteringPlan> clusteringPlan = table
+ .scheduleClustering(context, instantTime, extraMetadata);
+ option = clusteringPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case COMPACT:
+ LOG.info("Scheduling compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> compactionPlan = table
+ .scheduleCompaction(context, instantTime, extraMetadata);
+ option = compactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case LOG_COMPACT:
+ LOG.info("Scheduling log compaction at instant time :" + instantTime);
+ Option<HoodieCompactionPlan> logCompactionPlan = table
+ .scheduleLogCompaction(context, instantTime, extraMetadata);
+ option = logCompactionPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ case CLEAN:
+ LOG.info("Scheduling cleaning at instant time :" + instantTime);
+ Option<HoodieCleanerPlan> cleanerPlan = table
+ .scheduleCleaning(context, instantTime, extraMetadata);
+ option = cleanerPlan.isPresent() ? Option.of(instantTime) : Option.empty();
+ break;
+ default:
+ throw new IllegalArgumentException("Invalid TableService " + tableServiceType);
+ }
+
+ Option<String> instantRange = delegateToTableServiceManager(tableServiceType, table);
+ if (instantRange.isPresent()) {
+ LOG.info("Delegate instant [" + instantRange.get() + "] to table service manager");
+ }
+
+ return option;
+ }
+
+ protected abstract HoodieTable<?, ?, ?, ?> createTable(HoodieWriteConfig config, Configuration hadoopConf);
+
+ /**
+ * Executes a clustering plan on a table, serially before or after an insert/upsert action.
+ * Schedules and executes clustering inline.
+ */
+ protected Option<String> inlineClustering(Option<Map<String, String>> extraMetadata) {
+ Option<String> clusteringInstantOpt = inlineScheduleClustering(extraMetadata);
+ clusteringInstantOpt.ifPresent(clusteringInstant -> {
+ // inline cluster should auto commit as the user is never given control
+ cluster(clusteringInstant, true);
+ });
+ return clusteringInstantOpt;
+ }
+
+ private void inlineClustering(HoodieTable table, Option<Map<String, String>> extraMetadata) {
+ if (shouldDelegateToTableServiceManager(config, ActionType.replacecommit)) {
+ scheduleClustering(extraMetadata);
+ } else {
+ runAnyPendingClustering(table);
+ inlineClustering(extraMetadata);
+ }
+ }
+
+ /**
+ * Schedules clustering inline.
+ *
+ * @param extraMetadata extrametadata to use.
+ * @return clustering instant if scheduled.
+ */
+ protected Option<String> inlineScheduleClustering(Option<Map<String, String>> extraMetadata) {
+ return scheduleClustering(extraMetadata);
+ }
+
+ protected void runAnyPendingClustering(HoodieTable table) {
+ table.getActiveTimeline().filterPendingReplaceTimeline().getInstants().forEach(instant -> {
+ Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(table.getMetaClient(), instant);
+ if (instantPlan.isPresent()) {
+ LOG.info("Running pending clustering at instant " + instantPlan.get().getLeft());
+ cluster(instant.getTimestamp(), true);
+ }
+ });
+ }
+
+ /**
+ * Finalize Write operation.
+ *
+ * @param table HoodieTable
+ * @param instantTime Instant Time
+ * @param stats Hoodie Write Stat
+ */
+ protected void finalizeWrite(HoodieTable table, String instantTime, List<HoodieWriteStat> stats) {
Review Comment:
remark: finalizeWrite is still retained in base write client for commit()
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -352,182 +303,23 @@ public void commitLogCompaction(String logCompactionInstantTime, HoodieCommitMet
protected void completeLogCompaction(HoodieCommitMetadata metadata,
HoodieTable table,
String logCompactionCommitTime) {
- this.context.setJobStatus(this.getClass().getSimpleName(), "Collect log compaction write status and commit compaction");
- List<HoodieWriteStat> writeStats = metadata.getWriteStats();
final HoodieInstant logCompactionInstant = new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.LOG_COMPACTION_ACTION, logCompactionCommitTime);
- try {
- this.txnManager.beginTransaction(Option.of(logCompactionInstant), Option.empty());
- preCommit(logCompactionInstant, metadata);
- finalizeWrite(table, logCompactionCommitTime, writeStats);
- // commit to data table after committing to metadata table.
- updateTableMetadata(table, metadata, logCompactionInstant);
- LOG.info("Committing Log Compaction " + logCompactionCommitTime + ". Finished with result " + metadata);
- CompactHelpers.getInstance().completeInflightLogCompaction(table, logCompactionCommitTime, metadata);
- } finally {
- this.txnManager.endTransaction(Option.of(logCompactionInstant));
- }
- WriteMarkersFactory.get(config.getMarkersType(), table, logCompactionCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
- if (compactionTimer != null) {
- long durationInMs = metrics.getDurationInMs(compactionTimer.stop());
- HoodieActiveTimeline.parseDateFromInstantTimeSafely(logCompactionCommitTime).ifPresent(parsedInstant ->
- metrics.updateCommitMetrics(parsedInstant.getTime(), durationInMs, metadata, HoodieActiveTimeline.LOG_COMPACTION_ACTION)
- );
- }
- LOG.info("Log Compacted successfully on commit " + logCompactionCommitTime);
+ preCommit(logCompactionInstant, metadata);
+ tableServiceClient.completeLogCompaction(metadata, table, logCompactionCommitTime);
Review Comment:
preCommit was part of the transaction, now it happens before transaction starts. What is the impact here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org