You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "stream2000 (via GitHub)" <gi...@apache.org> on 2023/08/28 11:12:50 UTC
[GitHub] [hudi] stream2000 opened a new pull request, #9558: [HUDI-6481] Support run multi tables services in a single spark job
stream2000 opened a new pull request, #9558:
URL: https://github.com/apache/hudi/pull/9558
### Change Logs
Now we have HoodieMultiTableDeltaStreamer using spark to ingest multi tables into hudi, and we can also ingest multi tables using a single flink in some platforms like alicloud vvp. In such scenario we may want to run all tables services outside of the ingestion job to improve stability.
Currently, hudi provides a bunch of offline table service jobs but we still need to run them manually or through some schedule tools. This Pr provides a tool to run all table services in a single spark job
### Impact
provides a tool to run all table services in a single spark job
### Risk level (write none, low medium or high below)
none
### Documentation Update
Will update document after merge
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1696020828
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab46e341122b1fe70ebc85aa0b1a97204b2e7f89 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1703610657
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ffc02724376dc67f1d5426fc1d95cbf1725d0261 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1316898057
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
> The existing constructor already satisfy the needs.
Yes, however if we want to reuse existing constructor, we need pass the prop file path and the cmd line config from `HoodieMultiTableServicesMain` to `HoodieCleaner`. And we will parse the config(read props file and use cmd line config to override it) for every `TableServiceTask` , which is redundant. Current implementation will do the config parsing in `HoodieMultiTableServicesMain` only, and all the `TableServiceTask` will use the final config parsed by `HoodieMultiTableServicesMain`. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311199016
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,249 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.utilities.multitable.MultiTableServiceUtils.Constants.LOCAL_SPARK_MASTER;
+
+public class HoodieMultiTableServicesMain {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);
+ final Config cfg;
+ final TypedProperties props;
+
+ private final JavaSparkContext jsc;
+
+ private ScheduledExecutorService executorService;
+
+ private void batchRunTableServices(List<String> tablePaths) throws InterruptedException, ExecutionException {
+ ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);
+ List<CompletableFuture<Void>> futures = tablePaths.stream()
+ .map(basePath -> CompletableFuture.runAsync(
+ () -> MultiTableServiceUtils.buildTableServicePipeline(jsc, basePath, cfg, props).execute(),
+ executorService))
+ .collect(Collectors.toList());
+ CompletableFuture<?> allComplete =
+ CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));
+ CompletableFuture<?> anyException = new CompletableFuture<>();
+ for (CompletableFuture<?> future : futures) {
+ future.exceptionally((e) -> {
+ anyException.completeExceptionally(e);
+ return null;
+ });
+ }
+ try {
+ CompletableFuture.anyOf(allComplete, anyException).get();
+ } catch (ExecutionException ee) {
+ throw new ExecutionException("some table service failed", ee);
+ } finally {
+ executorService.shutdownNow();
+ }
+ }
+
+ private void streamRunTableServices(List<String> tablePaths) throws InterruptedException {
+ executorService = Executors.newScheduledThreadPool(cfg.poolSize);
+ for (String tablePath : tablePaths) {
+ TableServicePipeline pipeline = MultiTableServiceUtils.buildTableServicePipeline(jsc, tablePath, cfg, props);
+ executorService.scheduleAtFixedRate(pipeline::execute, 0, cfg.scheduleDelay, TimeUnit.MILLISECONDS);
+ }
+ executorService.awaitTermination(Long.MAX_VALUE, TimeUnit.MINUTES);
+ }
+
+ public void cancel() {
+ if (executorService != null) {
+ executorService.shutdown();
+ }
+ }
+
+ public HoodieMultiTableServicesMain(JavaSparkContext jsc, Config cfg) {
+ this.cfg = cfg;
+ this.jsc = jsc;
+ this.props = cfg.propsFilePath == null ? UtilHelpers.buildProperties(cfg.configs) : readConfigFromFileSystem(jsc, cfg);
+ }
+
+ public void startServices() throws ExecutionException, InterruptedException {
+ LOG.info("StartServices Config: " + cfg);
+ List<String> tablePaths;
+ if (cfg.autoDiscovering) {
+ // We support defining multi base paths
+ tablePaths = cfg.basePath.stream()
+ .filter(this::pathExists)
+ .flatMap(p -> MultiTableServiceUtils.findHoodieTablesUnderPath(jsc, p).stream())
+ .collect(Collectors.toList());
+ } else {
+ tablePaths = MultiTableServiceUtils.getTablesToBeIngestedFromProps(props);
+ }
+ LOG.info("All table paths: " + String.join(",", tablePaths));
+ if (cfg.batch) {
+ batchRunTableServices(tablePaths);
+ } else {
+ streamRunTableServices(tablePaths);
+ }
+ }
+
+ private TypedProperties readConfigFromFileSystem(JavaSparkContext jsc, Config cfg) {
+ return UtilHelpers.readConfig(jsc.hadoopConfiguration(), new Path(cfg.propsFilePath), cfg.configs).getProps(true);
+ }
+
+ private boolean pathExists(String path) {
+ try {
+ Path p = new Path(path);
+ FileSystem fs = p.getFileSystem(jsc.hadoopConfiguration());
+ return fs.exists(p);
+ } catch (IOException e) {
+ throw new HoodieIOException("Error checking path existing:", e);
+ }
+ }
+
+ /**
+ * Command line configs to run table services
+ */
+ public static class Config implements Serializable {
+ @Parameter(names = {"--base-path"}, description = "Base path for all the tables, this can be repeated",
+ required = true, splitter = IdentitySplitter.class)
+ public List<String> basePath = Collections.emptyList();
+
+ @Parameter(names = {"--auto", "-a"}, description = "Whether to discover hudi tables in the base path")
Review Comment:
sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1309021962
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java:
##########
@@ -0,0 +1,86 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
Review Comment:
In all the files you have double commented, it should just be
```
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java:
##########
@@ -0,0 +1,86 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.utilities.UtilHelpers;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class ArchiveTask extends TableServiceTask {
Review Comment:
add javadocs
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java:
##########
@@ -71,6 +71,13 @@ public HoodieClusteringJob(JavaSparkContext jsc, Config cfg) {
}
}
+ public HoodieClusteringJob(JavaSparkContext jsc, Config cfg, TypedProperties props) {
+ this.cfg = cfg;
+ this.jsc = jsc;
+ this.props = props;
+ this.metaClient = UtilHelpers.createMetaClient(jsc, cfg.basePath, true);
+ }
Review Comment:
Do we want to do the disable async cleaning and add lock options like the other constructor?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,249 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.utilities.multitable.MultiTableServiceUtils.Constants.LOCAL_SPARK_MASTER;
+
+public class HoodieMultiTableServicesMain {
Review Comment:
javadocs
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/multitable/TestHoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,304 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDReadClient;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieCleanConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieLayoutConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+import org.apache.hudi.table.action.commit.SparkBucketIndexPartitioner;
+import org.apache.hudi.table.storage.HoodieStorageLayout;
+import org.apache.hudi.testutils.providers.SparkProvider;
+import org.apache.hudi.utilities.HoodieCompactor;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.HoodieSparkKryoRegistrar$;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.SparkSession;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+
+class TestHoodieMultiTableServicesMain extends HoodieCommonTestHarness implements SparkProvider {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ArchiveTask.class);
+
+ protected boolean initialized = false;
+
+ private static SparkSession spark;
+ private static SQLContext sqlContext;
+ private static JavaSparkContext jsc;
+ private static HoodieSparkEngineContext context;
+
+ protected transient HoodieTestDataGenerator dataGen = null;
+
+ @BeforeEach
+ public void init() throws IOException, ExecutionException, InterruptedException {
+ boolean initialized = spark != null;
+ if (!initialized) {
+ SparkConf sparkConf = conf();
+ HoodieSparkKryoRegistrar$.MODULE$.register(sparkConf);
+ SparkRDDReadClient.addHoodieSupport(sparkConf);
+ spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ sqlContext = spark.sqlContext();
+ jsc = new JavaSparkContext(spark.sparkContext());
+ context = new HoodieSparkEngineContext(jsc);
+ }
+ initPath();
+ prepareData();
+ }
+
+ @Test
+ public void testRunAllServices() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ cfg.batch = true;
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ HoodieTableMetaClient metaClient2 = getMetaClient("table2");
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ main.startServices();
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify delta commits
+ Assertions.assertEquals(2, metaClient1.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ Assertions.assertEquals(2, metaClient2.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ // Verify replace commits
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ Assertions.assertEquals(4, metaClient2.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ @Test
+ public void testRunAllServicesForSingleTable() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ cfg.batch = true;
+ cfg.basePath = Collections.singletonList(metaClient1.getBasePath());
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ main.startServices();
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify delta commits
+ Assertions.assertEquals(2, metaClient1.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ // Verify replace commits
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ @Test
+ public void testStreamRunAllServices() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ new Thread(() -> {
+ try {
+ Thread.sleep(10000);
+ LOG.info("Shutdown the table services");
+ main.cancel();
+ } catch (InterruptedException e) {
+ //
Review Comment:
missing?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java:
##########
@@ -74,6 +74,13 @@ public HoodieCompactor(JavaSparkContext jsc, Config cfg) {
}
}
+ public HoodieCompactor(JavaSparkContext jsc, Config cfg, TypedProperties props) {
+ this.cfg = cfg;
+ this.jsc = jsc;
+ this.props = props;
+ this.metaClient = UtilHelpers.createMetaClient(jsc, cfg.basePath, true);
+ }
Review Comment:
Same question as in clustering job
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/multitable/TestHoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,304 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDReadClient;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieCleanConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieLayoutConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+import org.apache.hudi.table.action.commit.SparkBucketIndexPartitioner;
+import org.apache.hudi.table.storage.HoodieStorageLayout;
+import org.apache.hudi.testutils.providers.SparkProvider;
+import org.apache.hudi.utilities.HoodieCompactor;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.HoodieSparkKryoRegistrar$;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.SparkSession;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+
+class TestHoodieMultiTableServicesMain extends HoodieCommonTestHarness implements SparkProvider {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ArchiveTask.class);
+
+ protected boolean initialized = false;
+
+ private static SparkSession spark;
+ private static SQLContext sqlContext;
+ private static JavaSparkContext jsc;
+ private static HoodieSparkEngineContext context;
+
+ protected transient HoodieTestDataGenerator dataGen = null;
+
+ @BeforeEach
+ public void init() throws IOException, ExecutionException, InterruptedException {
+ boolean initialized = spark != null;
+ if (!initialized) {
+ SparkConf sparkConf = conf();
+ HoodieSparkKryoRegistrar$.MODULE$.register(sparkConf);
+ SparkRDDReadClient.addHoodieSupport(sparkConf);
+ spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ sqlContext = spark.sqlContext();
+ jsc = new JavaSparkContext(spark.sparkContext());
+ context = new HoodieSparkEngineContext(jsc);
+ }
+ initPath();
+ prepareData();
+ }
+
+ @Test
+ public void testRunAllServices() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ cfg.batch = true;
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ HoodieTableMetaClient metaClient2 = getMetaClient("table2");
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ main.startServices();
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify delta commits
+ Assertions.assertEquals(2, metaClient1.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ Assertions.assertEquals(2, metaClient2.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ // Verify replace commits
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ Assertions.assertEquals(4, metaClient2.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ @Test
+ public void testRunAllServicesForSingleTable() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ cfg.batch = true;
+ cfg.basePath = Collections.singletonList(metaClient1.getBasePath());
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ main.startServices();
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify delta commits
+ Assertions.assertEquals(2, metaClient1.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ // Verify replace commits
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ @Test
+ public void testStreamRunAllServices() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ new Thread(() -> {
+ try {
+ Thread.sleep(10000);
+ LOG.info("Shutdown the table services");
+ main.cancel();
+ } catch (InterruptedException e) {
+ //
+ }
+ }).start();
+ main.startServices();
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ HoodieTableMetaClient metaClient2 = getMetaClient("table2");
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ Assertions.assertEquals(4, metaClient2.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ private void prepareData() throws IOException {
+ initTestDataGenerator();
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ HoodieTableMetaClient metaClient2 = getMetaClient("table2");
+ String instant1 = HoodieInstantTimeGenerator.createNewInstantTime(0);
+ writeToTable(metaClient1.getBasePath(), instant1, false);
+ writeToTable(metaClient2.getBasePath(), instant1, false);
+ String instant2 = HoodieInstantTimeGenerator.createNewInstantTime(1);
+ writeToTable(metaClient1.getBasePath(), instant2, true);
+ writeToTable(metaClient2.getBasePath(), instant2, true);
+ Assertions.assertEquals(0, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ Assertions.assertEquals(0, metaClient2.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ }
+
+ private void writeToTable(String basePath, String instant, boolean update) throws IOException {
+ String tableName = "test";
+ HoodieWriteConfig.Builder writeConfigBuilder = getWriteConfigBuilder(basePath, tableName);
+ // enable files and bloom_filters on the regular write client
+ HoodieWriteConfig writeConfig = writeConfigBuilder.build();
+ // do one upsert with synchronous metadata update
+ SparkRDDWriteClient writeClient = new SparkRDDWriteClient(context, writeConfig);
+ List<HoodieRecord> records;
+ writeClient.startCommitWithTime(instant);
+ if (update) {
+ records = dataGen.generateUpdates(instant, 100);
+ } else {
+ records = dataGen.generateInserts(instant, 100);
+ }
+ JavaRDD<WriteStatus> result = writeClient.upsert(jsc.parallelize(records, 8), instant);
+ List<WriteStatus> statuses = result.collect();
+ assertNoWriteErrors(statuses);
+ }
+
+ private HoodieWriteConfig.Builder getWriteConfigBuilder(String basePath, String tableName) {
+ Properties properties = new Properties();
+ properties.setProperty(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_row_key");
+ return HoodieWriteConfig.newBuilder()
+ .withPath(basePath)
+ .withSchema(HoodieTestDataGenerator.TRIP_EXAMPLE_SCHEMA)
+ .withParallelism(4, 4)
+ .withBulkInsertParallelism(4)
+ .withFinalizeWriteParallelism(2)
+ .withProps(makeIndexConfig(HoodieIndex.IndexType.BUCKET))
Review Comment:
why bucket index? Just curious
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/multitable/TestHoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,304 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDReadClient;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieCleanConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieLayoutConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+import org.apache.hudi.table.action.commit.SparkBucketIndexPartitioner;
+import org.apache.hudi.table.storage.HoodieStorageLayout;
+import org.apache.hudi.testutils.providers.SparkProvider;
+import org.apache.hudi.utilities.HoodieCompactor;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.HoodieSparkKryoRegistrar$;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.SparkSession;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+
+class TestHoodieMultiTableServicesMain extends HoodieCommonTestHarness implements SparkProvider {
Review Comment:
javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1708693499
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672",
"triggerID" : "1708066642",
"triggerType" : "MANUAL"
}, {
"hash" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672)
* 31dabb979fc3029bad5537c56b34bf7e9ab9a808 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1708059314
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1695640880
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab46e341122b1fe70ebc85aa0b1a97204b2e7f89 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1695575691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab46e341122b1fe70ebc85aa0b1a97204b2e7f89 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1700552015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab46e341122b1fe70ebc85aa0b1a97204b2e7f89 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520)
* bb592b7615a5b20676747abfd6883bfd107cd1aa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1315550922
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.utilities.UtilHelpers;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Archive task to run in TableServicePipeline
Review Comment:
done
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.utilities.UtilHelpers;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Archive task to run in TableServicePipeline
+ * @see HoodieMultiTableServicesMain
+ */
+class ArchiveTask extends TableServiceTask {
+ private static final Logger LOG = LoggerFactory.getLogger(ArchiveTask.class);
+
+ @Override
+ void run() {
+ LOG.info("Run Archive with props: " + props);
+ HoodieWriteConfig hoodieCfg = HoodieWriteConfig.newBuilder().withPath(basePath).withProps(props).build();
+ try (SparkRDDWriteClient client = new SparkRDDWriteClient<>(new HoodieSparkEngineContext(jsc), hoodieCfg)) {
+ UtilHelpers.retry(retry, () -> {
+ client.archive();
+ return 0;
+ }, "Archive Failed");
+ }
+ }
+
+ public static Builder newBuilder() {
+ return new Builder();
+ }
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1709382899
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672",
"triggerID" : "1708066642",
"triggerType" : "MANUAL"
}, {
"hash" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676",
"triggerID" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"triggerType" : "PUSH"
}, {
"hash" : "b67857bbee604515b2da3889763292837d644a2c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b67857bbee604515b2da3889763292837d644a2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31dabb979fc3029bad5537c56b34bf7e9ab9a808 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676)
* b67857bbee604515b2da3889763292837d644a2c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1709388276
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672",
"triggerID" : "1708066642",
"triggerType" : "MANUAL"
}, {
"hash" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676",
"triggerID" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"triggerType" : "PUSH"
}, {
"hash" : "b67857bbee604515b2da3889763292837d644a2c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19693",
"triggerID" : "b67857bbee604515b2da3889763292837d644a2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31dabb979fc3029bad5537c56b34bf7e9ab9a808 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676)
* b67857bbee604515b2da3889763292837d644a2c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19693)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1709956376
Merging this as CI failed with other reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1702129218
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0a5621c43699e3cd636c99ef6cc048788f04459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573)
* 1640805e55e219b1c512bde9650849613c03e0b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311197513
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/multitable/TestHoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,304 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDReadClient;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieCleanConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieLayoutConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+import org.apache.hudi.table.action.commit.SparkBucketIndexPartitioner;
+import org.apache.hudi.table.storage.HoodieStorageLayout;
+import org.apache.hudi.testutils.providers.SparkProvider;
+import org.apache.hudi.utilities.HoodieCompactor;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.HoodieSparkKryoRegistrar$;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.SparkSession;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+
+class TestHoodieMultiTableServicesMain extends HoodieCommonTestHarness implements SparkProvider {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ArchiveTask.class);
+
+ protected boolean initialized = false;
+
+ private static SparkSession spark;
+ private static SQLContext sqlContext;
+ private static JavaSparkContext jsc;
+ private static HoodieSparkEngineContext context;
+
+ protected transient HoodieTestDataGenerator dataGen = null;
+
+ @BeforeEach
+ public void init() throws IOException, ExecutionException, InterruptedException {
+ boolean initialized = spark != null;
+ if (!initialized) {
+ SparkConf sparkConf = conf();
+ HoodieSparkKryoRegistrar$.MODULE$.register(sparkConf);
+ SparkRDDReadClient.addHoodieSupport(sparkConf);
+ spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ sqlContext = spark.sqlContext();
+ jsc = new JavaSparkContext(spark.sparkContext());
+ context = new HoodieSparkEngineContext(jsc);
+ }
+ initPath();
+ prepareData();
+ }
+
+ @Test
+ public void testRunAllServices() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ cfg.batch = true;
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ HoodieTableMetaClient metaClient2 = getMetaClient("table2");
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ main.startServices();
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify delta commits
+ Assertions.assertEquals(2, metaClient1.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ Assertions.assertEquals(2, metaClient2.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ // Verify replace commits
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ Assertions.assertEquals(4, metaClient2.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ @Test
+ public void testRunAllServicesForSingleTable() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ cfg.batch = true;
+ cfg.basePath = Collections.singletonList(metaClient1.getBasePath());
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ main.startServices();
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify delta commits
+ Assertions.assertEquals(2, metaClient1.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ // Verify replace commits
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ @Test
+ public void testStreamRunAllServices() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ new Thread(() -> {
+ try {
+ Thread.sleep(10000);
+ LOG.info("Shutdown the table services");
+ main.cancel();
+ } catch (InterruptedException e) {
+ //
Review Comment:
updated and ignore the exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1700713447
We have already used this feature internally to make table service management of Hudi tables easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1696650479
@leesf @jonvex Hi, could you please help review this pr? Thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1316898057
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
> The existing constructor already satisfy the needs.
Yes, however if we want to reuse existing constructor, we need pass the prop file path and the cmd line conifig from `HoodieMultiTableServicesMain` to `HoodieCleaner`. And we will parse the config for every `TableServiceTask` , which is redundant. Current implementation will do the config parsing in `HoodieMultiTableServicesMain` only, and all the `TableServiceTask` will use the final config parsed by `HoodieMultiTableServicesMain`. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1708708018
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672",
"triggerID" : "1708066642",
"triggerType" : "MANUAL"
}, {
"hash" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676",
"triggerID" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672)
* 31dabb979fc3029bad5537c56b34bf7e9ab9a808 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1317960979
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
Fine with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1709918900
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672",
"triggerID" : "1708066642",
"triggerType" : "MANUAL"
}, {
"hash" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676",
"triggerID" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"triggerType" : "PUSH"
}, {
"hash" : "b67857bbee604515b2da3889763292837d644a2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19693",
"triggerID" : "b67857bbee604515b2da3889763292837d644a2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b67857bbee604515b2da3889763292837d644a2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19693)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311178312
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/multitable/TestHoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,304 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDReadClient;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieCleanConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.config.HoodieLayoutConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+import org.apache.hudi.table.action.commit.SparkBucketIndexPartitioner;
+import org.apache.hudi.table.storage.HoodieStorageLayout;
+import org.apache.hudi.testutils.providers.SparkProvider;
+import org.apache.hudi.utilities.HoodieCompactor;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.HoodieSparkKryoRegistrar$;
+import org.apache.spark.SparkConf;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.SQLContext;
+import org.apache.spark.sql.SparkSession;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+
+class TestHoodieMultiTableServicesMain extends HoodieCommonTestHarness implements SparkProvider {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ArchiveTask.class);
+
+ protected boolean initialized = false;
+
+ private static SparkSession spark;
+ private static SQLContext sqlContext;
+ private static JavaSparkContext jsc;
+ private static HoodieSparkEngineContext context;
+
+ protected transient HoodieTestDataGenerator dataGen = null;
+
+ @BeforeEach
+ public void init() throws IOException, ExecutionException, InterruptedException {
+ boolean initialized = spark != null;
+ if (!initialized) {
+ SparkConf sparkConf = conf();
+ HoodieSparkKryoRegistrar$.MODULE$.register(sparkConf);
+ SparkRDDReadClient.addHoodieSupport(sparkConf);
+ spark = SparkSession.builder().config(sparkConf).getOrCreate();
+ sqlContext = spark.sqlContext();
+ jsc = new JavaSparkContext(spark.sparkContext());
+ context = new HoodieSparkEngineContext(jsc);
+ }
+ initPath();
+ prepareData();
+ }
+
+ @Test
+ public void testRunAllServices() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ cfg.batch = true;
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ HoodieTableMetaClient metaClient2 = getMetaClient("table2");
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ main.startServices();
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify delta commits
+ Assertions.assertEquals(2, metaClient1.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ Assertions.assertEquals(2, metaClient2.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ // Verify replace commits
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ Assertions.assertEquals(4, metaClient2.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ @Test
+ public void testRunAllServicesForSingleTable() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ cfg.batch = true;
+ cfg.basePath = Collections.singletonList(metaClient1.getBasePath());
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ main.startServices();
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify delta commits
+ Assertions.assertEquals(2, metaClient1.reloadActiveTimeline().getDeltaCommitTimeline().countInstants());
+ // Verify replace commits
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCompletedReplaceTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ @Test
+ public void testStreamRunAllServices() throws IOException, ExecutionException, InterruptedException {
+ HoodieMultiTableServicesMain.Config cfg = getHoodieMultiServiceConfig();
+ HoodieMultiTableServicesMain main = new HoodieMultiTableServicesMain(jsc, cfg);
+ new Thread(() -> {
+ try {
+ Thread.sleep(10000);
+ LOG.info("Shutdown the table services");
+ main.cancel();
+ } catch (InterruptedException e) {
+ //
+ }
+ }).start();
+ main.startServices();
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ HoodieTableMetaClient metaClient2 = getMetaClient("table2");
+ // Verify cleans
+ Assertions.assertEquals(1, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ Assertions.assertEquals(1, metaClient2.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ // Verify compactions, delta commits and replace commits
+ Assertions.assertEquals(4, metaClient1.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ Assertions.assertEquals(4, metaClient2.reloadActiveTimeline().getCommitsTimeline().countInstants());
+ }
+
+ private void prepareData() throws IOException {
+ initTestDataGenerator();
+ HoodieTableMetaClient metaClient1 = getMetaClient("table1");
+ HoodieTableMetaClient metaClient2 = getMetaClient("table2");
+ String instant1 = HoodieInstantTimeGenerator.createNewInstantTime(0);
+ writeToTable(metaClient1.getBasePath(), instant1, false);
+ writeToTable(metaClient2.getBasePath(), instant1, false);
+ String instant2 = HoodieInstantTimeGenerator.createNewInstantTime(1);
+ writeToTable(metaClient1.getBasePath(), instant2, true);
+ writeToTable(metaClient2.getBasePath(), instant2, true);
+ Assertions.assertEquals(0, metaClient1.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ Assertions.assertEquals(0, metaClient2.reloadActiveTimeline().getCleanerTimeline().countInstants());
+ }
+
+ private void writeToTable(String basePath, String instant, boolean update) throws IOException {
+ String tableName = "test";
+ HoodieWriteConfig.Builder writeConfigBuilder = getWriteConfigBuilder(basePath, tableName);
+ // enable files and bloom_filters on the regular write client
+ HoodieWriteConfig writeConfig = writeConfigBuilder.build();
+ // do one upsert with synchronous metadata update
+ SparkRDDWriteClient writeClient = new SparkRDDWriteClient(context, writeConfig);
+ List<HoodieRecord> records;
+ writeClient.startCommitWithTime(instant);
+ if (update) {
+ records = dataGen.generateUpdates(instant, 100);
+ } else {
+ records = dataGen.generateInserts(instant, 100);
+ }
+ JavaRDD<WriteStatus> result = writeClient.upsert(jsc.parallelize(records, 8), instant);
+ List<WriteStatus> statuses = result.collect();
+ assertNoWriteErrors(statuses);
+ }
+
+ private HoodieWriteConfig.Builder getWriteConfigBuilder(String basePath, String tableName) {
+ Properties properties = new Properties();
+ properties.setProperty(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_row_key");
+ return HoodieWriteConfig.newBuilder()
+ .withPath(basePath)
+ .withSchema(HoodieTestDataGenerator.TRIP_EXAMPLE_SCHEMA)
+ .withParallelism(4, 4)
+ .withBulkInsertParallelism(4)
+ .withFinalizeWriteParallelism(2)
+ .withProps(makeIndexConfig(HoodieIndex.IndexType.BUCKET))
Review Comment:
Just want to make it easier to produce log for compaciton.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311198668
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java:
##########
@@ -0,0 +1,166 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.table.HoodieTableMetaClient.METAFOLDER_NAME;
+
+public class MultiTableServiceUtils {
+
+ public static class Constants {
+ public static final String TABLES_TO_BE_INGESTED_PROP = "hoodie.tableservice.tablesToServe";
+
+ public static final String COMMA_SEPARATOR = ",";
+
+ public static final String LOCAL_SPARK_MASTER = "local[2]";
Review Comment:
move it to cli param just as other standalone table service job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311190312
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,249 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.utilities.multitable.MultiTableServiceUtils.Constants.LOCAL_SPARK_MASTER;
+
+public class HoodieMultiTableServicesMain {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);
+ final Config cfg;
+ final TypedProperties props;
+
+ private final JavaSparkContext jsc;
+
+ private ScheduledExecutorService executorService;
+
+ private void batchRunTableServices(List<String> tablePaths) throws InterruptedException, ExecutionException {
+ ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);
Review Comment:
I don't really understand what you mean here? I created a new local thread pool for running all table services and destroyed it since we only need it once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311188286
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java:
##########
@@ -0,0 +1,86 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.utilities.UtilHelpers;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+class ArchiveTask extends TableServiceTask {
Review Comment:
Added javadoc for all new classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1706168245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84d10c95a1f6b5395a85014031ac32267ac42fbc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623)
* aa20e8bfd05272fa8cffeff00d7b3612d2ef7081 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1706239234
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84d10c95a1f6b5395a85014031ac32267ac42fbc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623)
* aa20e8bfd05272fa8cffeff00d7b3612d2ef7081 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1706269598
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84d10c95a1f6b5395a85014031ac32267ac42fbc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623)
* aa20e8bfd05272fa8cffeff00d7b3612d2ef7081 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650)
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf merged pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf merged PR #9558:
URL: https://github.com/apache/hudi/pull/9558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1316898057
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
> The existing constructor already satisfy the needs.
Yes, however if we want to reuse existing constructor, we need pass the prop file path and the cmd line config from `HoodieMultiTableServicesMain` to `HoodieCleaner`. And we will parse the config for every `TableServiceTask` , which is redundant. Current implementation will do the config parsing in `HoodieMultiTableServicesMain` only, and all the `TableServiceTask` will use the final config parsed by `HoodieMultiTableServicesMain`. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1708066642
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1709045064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672",
"triggerID" : "1708066642",
"triggerType" : "MANUAL"
}, {
"hash" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676",
"triggerID" : "31dabb979fc3029bad5537c56b34bf7e9ab9a808",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31dabb979fc3029bad5537c56b34bf7e9ab9a808 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19676)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311187301
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java:
##########
@@ -71,6 +71,13 @@ public HoodieClusteringJob(JavaSparkContext jsc, Config cfg) {
}
}
+ public HoodieClusteringJob(JavaSparkContext jsc, Config cfg, TypedProperties props) {
+ this.cfg = cfg;
+ this.jsc = jsc;
+ this.props = props;
+ this.metaClient = UtilHelpers.createMetaClient(jsc, cfg.basePath, true);
+ }
Review Comment:
Sure! Thanks for reminding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1700564784
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab46e341122b1fe70ebc85aa0b1a97204b2e7f89 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520)
* bb592b7615a5b20676747abfd6883bfd107cd1aa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572)
* d0a5621c43699e3cd636c99ef6cc048788f04459 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1317999559
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
> Then we can make one constructor to invokes the other one.
Thanks for your suggestions~ I have slightly refactored the code and make it cleaner, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1706337461
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa20e8bfd05272fa8cffeff00d7b3612d2ef7081 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650)
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1706633047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1708072951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672",
"triggerID" : "1708066642",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1317961289
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
Then we can force one constructor to invoke the other one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1702740545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffc02724376dc67f1d5426fc1d95cbf1725d0261 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1315315332
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.utilities.UtilHelpers;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Archive task to run in TableServicePipeline
+ * @see HoodieMultiTableServicesMain
+ */
+class ArchiveTask extends TableServiceTask {
+ private static final Logger LOG = LoggerFactory.getLogger(ArchiveTask.class);
+
+ @Override
+ void run() {
+ LOG.info("Run Archive with props: " + props);
+ HoodieWriteConfig hoodieCfg = HoodieWriteConfig.newBuilder().withPath(basePath).withProps(props).build();
+ try (SparkRDDWriteClient client = new SparkRDDWriteClient<>(new HoodieSparkEngineContext(jsc), hoodieCfg)) {
+ UtilHelpers.retry(retry, () -> {
+ client.archive();
+ return 0;
+ }, "Archive Failed");
+ }
+ }
+
+ public static Builder newBuilder() {
+ return new Builder();
+ }
+
Review Comment:
Add doc for each `Builder` class, also the doc for each member variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1708499358
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672",
"triggerID" : "1708066642",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19672)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1700578387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bb592b7615a5b20676747abfd6883bfd107cd1aa Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572)
* d0a5621c43699e3cd636c99ef6cc048788f04459 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1317961289
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
Then we can make one constructor to invokes the other one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1702225241
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1640805e55e219b1c512bde9650849613c03e0b9 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598)
* ffc02724376dc67f1d5426fc1d95cbf1725d0261 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1315399483
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
Yes in multi-table services we also fetch the props either from cmd line or global options. And we do not want each service to fetch the props by themselves. So here I add a new constructor to use the props that all services shared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1312487436
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.table.HoodieTableMetaClient.METAFOLDER_NAME;
+
+/**
+ * Utils for executing multi-table services
+ */
+public class MultiTableServiceUtils {
+
+ public static class Constants {
+ public static final String TABLES_TO_BE_SERVED_PROP = "hoodie.tableservice.tablesToServe";
+
+ public static final String COMMA_SEPARATOR = ",";
+
+ private static final int DEFAULT_LISTING_PARALLELISM = 1500;
+ }
+
+ public static List<String> getTablesToBeServedFromProps(TypedProperties properties) {
+ String combinedTablesString = properties.getString(Constants.TABLES_TO_BE_SERVED_PROP);
+ if (combinedTablesString == null) {
+ return new ArrayList<>();
+ }
+ String[] tablesArray = combinedTablesString.split(Constants.COMMA_SEPARATOR);
+ return Arrays.asList(tablesArray);
+ }
+
+ public static List<String> findHoodieTablesUnderPath(JavaSparkContext jsc, String pathStr) {
+ Path rootPath = new Path(pathStr);
+ SerializableConfiguration conf = new SerializableConfiguration(jsc.hadoopConfiguration());
+ if (isHoodieTable(rootPath, conf.get())) {
+ return Collections.singletonList(pathStr);
+ }
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ List<String> hoodieTablePaths = new CopyOnWriteArrayList<>();
+ List<Path> pathsToList = new CopyOnWriteArrayList<>();
+ pathsToList.add(rootPath);
+ int listingParallelism = Math.min(Constants.DEFAULT_LISTING_PARALLELISM, pathsToList.size());
+
+ while (!pathsToList.isEmpty()) {
+ // List all directories in parallel
+ List<FileStatus[]> dirToFileListing = engineContext.map(pathsToList, path -> {
+ FileSystem fileSystem = path.getFileSystem(conf.get());
+ return fileSystem.listStatus(path);
+ }, listingParallelism);
+ pathsToList.clear();
+
+ // if current dictionary contains meta folder(.hoodie), add it to result. Otherwise, add it to queue
+ List<FileStatus> dirs = dirToFileListing.stream().flatMap(Arrays::stream)
+ .filter(FileStatus::isDirectory)
+ .collect(Collectors.toList());
+
+ if (!dirs.isEmpty()) {
+ List<Pair<FileStatus, Integer>> dirResults = engineContext.map(dirs, fileStatus -> {
+ if (isHoodieTable(fileStatus.getPath(), conf.get())) {
Review Comment:
Nice catch~ It's not a good design that uses hard-coded magic number, already updated the magic number to meaningful enum constants.
Updated to:
```java
/**
* Type of directories when searching hoodie tables under path
*/
enum DirType {
HOODIE_TABLE, // previous 0
NORMAL_DIR, // previous 1
META_FOLDER // previous 2
}
```
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.StringJoiner;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * Main function for executing multi-table services
+ */
+public class HoodieMultiTableServicesMain {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);
+ final Config cfg;
+ final TypedProperties props;
+
+ private final JavaSparkContext jsc;
+
+ private ScheduledExecutorService executorService;
+
+ private void batchRunTableServices(List<String> tablePaths) throws InterruptedException, ExecutionException {
+ ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);
+ List<CompletableFuture<Void>> futures = tablePaths.stream()
+ .map(basePath -> CompletableFuture.runAsync(
+ () -> MultiTableServiceUtils.buildTableServicePipeline(jsc, basePath, cfg, props).execute(),
Review Comment:
If no services is enabled, the `TableServicePipeline` will contain no task and the execute method will return directly. So it's OK to keep current design.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.StringJoiner;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * Main function for executing multi-table services
+ */
+public class HoodieMultiTableServicesMain {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);
Review Comment:
Sure, I missed it when migrated it from `HoodieStreamer`. Will check other classes also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1703549054
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1700491685
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab46e341122b1fe70ebc85aa0b1a97204b2e7f89 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520)
* bb592b7615a5b20676747abfd6883bfd107cd1aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1703890551
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84d10c95a1f6b5395a85014031ac32267ac42fbc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1703699610
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ffc02724376dc67f1d5426fc1d95cbf1725d0261 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1312423032
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.StringJoiner;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * Main function for executing multi-table services
+ */
+public class HoodieMultiTableServicesMain {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);
+ final Config cfg;
+ final TypedProperties props;
+
+ private final JavaSparkContext jsc;
+
+ private ScheduledExecutorService executorService;
+
+ private void batchRunTableServices(List<String> tablePaths) throws InterruptedException, ExecutionException {
+ ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);
+ List<CompletableFuture<Void>> futures = tablePaths.stream()
+ .map(basePath -> CompletableFuture.runAsync(
+ () -> MultiTableServiceUtils.buildTableServicePipeline(jsc, basePath, cfg, props).execute(),
Review Comment:
should early exit if no services is enabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311649118
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.table.HoodieTableMetaClient.METAFOLDER_NAME;
+
+/**
+ * Utils for executing multi-table services
+ */
+public class MultiTableServiceUtils {
+
+ public static class Constants {
+ public static final String TABLES_TO_BE_SERVED_PROP = "hoodie.tableservice.tablesToServe";
+
+ public static final String COMMA_SEPARATOR = ",";
+
+ private static final int DEFAULT_LISTING_PARALLELISM = 1500;
+ }
+
+ public static List<String> getTablesToBeServedFromProps(TypedProperties properties) {
+ String combinedTablesString = properties.getString(Constants.TABLES_TO_BE_SERVED_PROP);
+ if (combinedTablesString == null) {
+ return new ArrayList<>();
+ }
+ String[] tablesArray = combinedTablesString.split(Constants.COMMA_SEPARATOR);
+ return Arrays.asList(tablesArray);
+ }
+
+ public static List<String> findHoodieTablesUnderPath(JavaSparkContext jsc, String pathStr) {
+ Path rootPath = new Path(pathStr);
+ SerializableConfiguration conf = new SerializableConfiguration(jsc.hadoopConfiguration());
+ if (isHoodieTable(rootPath, conf.get())) {
+ return Collections.singletonList(pathStr);
+ }
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ List<String> hoodieTablePaths = new CopyOnWriteArrayList<>();
+ List<Path> pathsToList = new CopyOnWriteArrayList<>();
+ pathsToList.add(rootPath);
+ int listingParallelism = Math.min(Constants.DEFAULT_LISTING_PARALLELISM, pathsToList.size());
+
+ while (!pathsToList.isEmpty()) {
+ // List all directories in parallel
+ List<FileStatus[]> dirToFileListing = engineContext.map(pathsToList, path -> {
+ FileSystem fileSystem = path.getFileSystem(conf.get());
+ return fileSystem.listStatus(path);
+ }, listingParallelism);
+ pathsToList.clear();
+
+ // if current dictionary contains meta folder(.hoodie), add it to result. Otherwise, add it to queue
+ List<FileStatus> dirs = dirToFileListing.stream().flatMap(Arrays::stream)
+ .filter(FileStatus::isDirectory)
+ .collect(Collectors.toList());
+
+ if (!dirs.isEmpty()) {
+ List<Pair<FileStatus, Integer>> dirResults = engineContext.map(dirs, fileStatus -> {
+ if (isHoodieTable(fileStatus.getPath(), conf.get())) {
Review Comment:
what does 0, 1, 2 mean, looks like magic number?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.table.HoodieTableMetaClient.METAFOLDER_NAME;
+
+/**
+ * Utils for executing multi-table services
+ */
+public class MultiTableServiceUtils {
+
+ public static class Constants {
+ public static final String TABLES_TO_BE_SERVED_PROP = "hoodie.tableservice.tablesToServe";
+
+ public static final String COMMA_SEPARATOR = ",";
+
+ private static final int DEFAULT_LISTING_PARALLELISM = 1500;
+ }
+
+ public static List<String> getTablesToBeServedFromProps(TypedProperties properties) {
+ String combinedTablesString = properties.getString(Constants.TABLES_TO_BE_SERVED_PROP);
+ if (combinedTablesString == null) {
+ return new ArrayList<>();
+ }
+ String[] tablesArray = combinedTablesString.split(Constants.COMMA_SEPARATOR);
+ return Arrays.asList(tablesArray);
+ }
+
+ public static List<String> findHoodieTablesUnderPath(JavaSparkContext jsc, String pathStr) {
+ Path rootPath = new Path(pathStr);
+ SerializableConfiguration conf = new SerializableConfiguration(jsc.hadoopConfiguration());
+ if (isHoodieTable(rootPath, conf.get())) {
+ return Collections.singletonList(pathStr);
+ }
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ List<String> hoodieTablePaths = new CopyOnWriteArrayList<>();
+ List<Path> pathsToList = new CopyOnWriteArrayList<>();
+ pathsToList.add(rootPath);
+ int listingParallelism = Math.min(Constants.DEFAULT_LISTING_PARALLELISM, pathsToList.size());
+
+ while (!pathsToList.isEmpty()) {
+ // List all directories in parallel
+ List<FileStatus[]> dirToFileListing = engineContext.map(pathsToList, path -> {
+ FileSystem fileSystem = path.getFileSystem(conf.get());
+ return fileSystem.listStatus(path);
+ }, listingParallelism);
+ pathsToList.clear();
+
+ // if current dictionary contains meta folder(.hoodie), add it to result. Otherwise, add it to queue
+ List<FileStatus> dirs = dirToFileListing.stream().flatMap(Arrays::stream)
+ .filter(FileStatus::isDirectory)
+ .collect(Collectors.toList());
+
+ if (!dirs.isEmpty()) {
+ List<Pair<FileStatus, Integer>> dirResults = engineContext.map(dirs, fileStatus -> {
+ if (isHoodieTable(fileStatus.getPath(), conf.get())) {
Review Comment:
what does 0, 1, 2 mean? looks like magic number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1702134207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0a5621c43699e3cd636c99ef6cc048788f04459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573)
* 1640805e55e219b1c512bde9650849613c03e0b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311649118
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.table.HoodieTableMetaClient.METAFOLDER_NAME;
+
+/**
+ * Utils for executing multi-table services
+ */
+public class MultiTableServiceUtils {
+
+ public static class Constants {
+ public static final String TABLES_TO_BE_SERVED_PROP = "hoodie.tableservice.tablesToServe";
+
+ public static final String COMMA_SEPARATOR = ",";
+
+ private static final int DEFAULT_LISTING_PARALLELISM = 1500;
+ }
+
+ public static List<String> getTablesToBeServedFromProps(TypedProperties properties) {
+ String combinedTablesString = properties.getString(Constants.TABLES_TO_BE_SERVED_PROP);
+ if (combinedTablesString == null) {
+ return new ArrayList<>();
+ }
+ String[] tablesArray = combinedTablesString.split(Constants.COMMA_SEPARATOR);
+ return Arrays.asList(tablesArray);
+ }
+
+ public static List<String> findHoodieTablesUnderPath(JavaSparkContext jsc, String pathStr) {
+ Path rootPath = new Path(pathStr);
+ SerializableConfiguration conf = new SerializableConfiguration(jsc.hadoopConfiguration());
+ if (isHoodieTable(rootPath, conf.get())) {
+ return Collections.singletonList(pathStr);
+ }
+
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ List<String> hoodieTablePaths = new CopyOnWriteArrayList<>();
+ List<Path> pathsToList = new CopyOnWriteArrayList<>();
+ pathsToList.add(rootPath);
+ int listingParallelism = Math.min(Constants.DEFAULT_LISTING_PARALLELISM, pathsToList.size());
+
+ while (!pathsToList.isEmpty()) {
+ // List all directories in parallel
+ List<FileStatus[]> dirToFileListing = engineContext.map(pathsToList, path -> {
+ FileSystem fileSystem = path.getFileSystem(conf.get());
+ return fileSystem.listStatus(path);
+ }, listingParallelism);
+ pathsToList.clear();
+
+ // if current dictionary contains meta folder(.hoodie), add it to result. Otherwise, add it to queue
+ List<FileStatus> dirs = dirToFileListing.stream().flatMap(Arrays::stream)
+ .filter(FileStatus::isDirectory)
+ .collect(Collectors.toList());
+
+ if (!dirs.isEmpty()) {
+ List<Pair<FileStatus, Integer>> dirResults = engineContext.map(dirs, fileStatus -> {
+ if (isHoodieTable(fileStatus.getPath(), conf.get())) {
Review Comment:
what does 0, 1, 2 means, looks like magic number?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311644056
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.StringJoiner;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * Main function for executing multi-table services
+ */
+public class HoodieMultiTableServicesMain {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);
Review Comment:
HoodieMultiTableServicesMain.class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1315315003
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.utilities.UtilHelpers;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Archive task to run in TableServicePipeline
Review Comment:
Every doc sentence should end up with dot `.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1704517968
@leesf @jonvex Hi, thanks for your review! I have addressed all comments and can we move on this~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1703846562
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffc02724376dc67f1d5426fc1d95cbf1725d0261 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615)
* 84d10c95a1f6b5395a85014031ac32267ac42fbc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1707736446
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19623",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
}, {
"hash" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19650",
"triggerID" : "aa20e8bfd05272fa8cffeff00d7b3612d2ef7081",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651",
"triggerID" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"triggerType" : "PUSH"
}, {
"hash" : "e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668",
"triggerID" : "1707700226",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e7a6ae7c21c11b8df7b3f6885439a5a7ff0cb289 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19651) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19668)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1697746599
@stream2000 also please [sign up for jira and email the dev mailing list](https://hudi.apache.org/contribute/how-to-contribute) if you haven't. And assign yourself to the [jira ticket](https://issues.apache.org/jira/browse/HUDI-6481).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1309020323
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/TableServicePipeline.java:
##########
@@ -0,0 +1,40 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class TableServicePipeline {
Review Comment:
javadoc
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/MultiTableServiceUtils.java:
##########
@@ -0,0 +1,166 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.table.HoodieTableMetaClient.METAFOLDER_NAME;
+
+public class MultiTableServiceUtils {
+
+ public static class Constants {
+ public static final String TABLES_TO_BE_INGESTED_PROP = "hoodie.tableservice.tablesToServe";
+
+ public static final String COMMA_SEPARATOR = ",";
+
+ public static final String LOCAL_SPARK_MASTER = "local[2]";
Review Comment:
not sure if we want this hardcoded
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,249 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.utilities.multitable.MultiTableServiceUtils.Constants.LOCAL_SPARK_MASTER;
+
+public class HoodieMultiTableServicesMain {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);
+ final Config cfg;
+ final TypedProperties props;
+
+ private final JavaSparkContext jsc;
+
+ private ScheduledExecutorService executorService;
+
+ private void batchRunTableServices(List<String> tablePaths) throws InterruptedException, ExecutionException {
+ ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);
Review Comment:
do you want to create a local variable here? I think no, so you can use your cancel method
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ClusteringTask.java:
##########
@@ -0,0 +1,97 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.utilities.HoodieClusteringJob;
+
+import org.apache.spark.api.java.JavaSparkContext;
+
+class ClusteringTask extends TableServiceTask {
Review Comment:
javadocs
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/CompactionTask.java:
##########
@@ -0,0 +1,100 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy;
+import org.apache.hudi.utilities.HoodieCompactor;
+
+import org.apache.spark.api.java.JavaSparkContext;
+
+class CompactionTask extends TableServiceTask {
Review Comment:
javadocs
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/CleanTask.java:
##########
@@ -0,0 +1,79 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.utilities.HoodieCleaner;
+import org.apache.hudi.utilities.UtilHelpers;
+
+import org.apache.spark.api.java.JavaSparkContext;
+
+class CleanTask extends TableServiceTask {
Review Comment:
javadocs
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/HoodieMultiTableServicesMain.java:
##########
@@ -0,0 +1,249 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
+
+package org.apache.hudi.utilities.multitable;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.utilities.HoodieCompactor;
+import org.apache.hudi.utilities.IdentitySplitter;
+import org.apache.hudi.utilities.UtilHelpers;
+import org.apache.hudi.utilities.streamer.HoodieStreamer;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.utilities.multitable.MultiTableServiceUtils.Constants.LOCAL_SPARK_MASTER;
+
+public class HoodieMultiTableServicesMain {
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieStreamer.class);
+ final Config cfg;
+ final TypedProperties props;
+
+ private final JavaSparkContext jsc;
+
+ private ScheduledExecutorService executorService;
+
+ private void batchRunTableServices(List<String> tablePaths) throws InterruptedException, ExecutionException {
+ ExecutorService executorService = Executors.newFixedThreadPool(cfg.poolSize);
+ List<CompletableFuture<Void>> futures = tablePaths.stream()
+ .map(basePath -> CompletableFuture.runAsync(
+ () -> MultiTableServiceUtils.buildTableServicePipeline(jsc, basePath, cfg, props).execute(),
+ executorService))
+ .collect(Collectors.toList());
+ CompletableFuture<?> allComplete =
+ CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));
+ CompletableFuture<?> anyException = new CompletableFuture<>();
+ for (CompletableFuture<?> future : futures) {
+ future.exceptionally((e) -> {
+ anyException.completeExceptionally(e);
+ return null;
+ });
+ }
+ try {
+ CompletableFuture.anyOf(allComplete, anyException).get();
+ } catch (ExecutionException ee) {
+ throw new ExecutionException("some table service failed", ee);
+ } finally {
+ executorService.shutdownNow();
+ }
+ }
+
+ private void streamRunTableServices(List<String> tablePaths) throws InterruptedException {
+ executorService = Executors.newScheduledThreadPool(cfg.poolSize);
+ for (String tablePath : tablePaths) {
+ TableServicePipeline pipeline = MultiTableServiceUtils.buildTableServicePipeline(jsc, tablePath, cfg, props);
+ executorService.scheduleAtFixedRate(pipeline::execute, 0, cfg.scheduleDelay, TimeUnit.MILLISECONDS);
+ }
+ executorService.awaitTermination(Long.MAX_VALUE, TimeUnit.MINUTES);
+ }
+
+ public void cancel() {
+ if (executorService != null) {
+ executorService.shutdown();
+ }
+ }
+
+ public HoodieMultiTableServicesMain(JavaSparkContext jsc, Config cfg) {
+ this.cfg = cfg;
+ this.jsc = jsc;
+ this.props = cfg.propsFilePath == null ? UtilHelpers.buildProperties(cfg.configs) : readConfigFromFileSystem(jsc, cfg);
+ }
+
+ public void startServices() throws ExecutionException, InterruptedException {
+ LOG.info("StartServices Config: " + cfg);
+ List<String> tablePaths;
+ if (cfg.autoDiscovering) {
+ // We support defining multi base paths
+ tablePaths = cfg.basePath.stream()
+ .filter(this::pathExists)
+ .flatMap(p -> MultiTableServiceUtils.findHoodieTablesUnderPath(jsc, p).stream())
+ .collect(Collectors.toList());
+ } else {
+ tablePaths = MultiTableServiceUtils.getTablesToBeIngestedFromProps(props);
+ }
+ LOG.info("All table paths: " + String.join(",", tablePaths));
+ if (cfg.batch) {
+ batchRunTableServices(tablePaths);
+ } else {
+ streamRunTableServices(tablePaths);
+ }
+ }
+
+ private TypedProperties readConfigFromFileSystem(JavaSparkContext jsc, Config cfg) {
+ return UtilHelpers.readConfig(jsc.hadoopConfiguration(), new Path(cfg.propsFilePath), cfg.configs).getProps(true);
+ }
+
+ private boolean pathExists(String path) {
+ try {
+ Path p = new Path(path);
+ FileSystem fs = p.getFileSystem(jsc.hadoopConfiguration());
+ return fs.exists(p);
+ } catch (IOException e) {
+ throw new HoodieIOException("Error checking path existing:", e);
+ }
+ }
+
+ /**
+ * Command line configs to run table services
+ */
+ public static class Config implements Serializable {
+ @Parameter(names = {"--base-path"}, description = "Base path for all the tables, this can be repeated",
+ required = true, splitter = IdentitySplitter.class)
+ public List<String> basePath = Collections.emptyList();
+
+ @Parameter(names = {"--auto", "-a"}, description = "Whether to discover hudi tables in the base path")
Review Comment:
maybe "--auto-discovery"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311188021
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactor.java:
##########
@@ -74,6 +74,13 @@ public HoodieCompactor(JavaSparkContext jsc, Config cfg) {
}
}
+ public HoodieCompactor(JavaSparkContext jsc, Config cfg, TypedProperties props) {
+ this.cfg = cfg;
+ this.jsc = jsc;
+ this.props = props;
+ this.metaClient = UtilHelpers.createMetaClient(jsc, cfg.basePath, true);
+ }
Review Comment:
sure and updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1311197963
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/multitable/ArchiveTask.java:
##########
@@ -0,0 +1,86 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one or more
+ * * contributor license agreements. See the NOTICE file distributed with
+ * * this work for additional information regarding copyright ownership.
+ * * The ASF licenses this file to You under the Apache License, Version 2.0
+ * * (the "License"); you may not use this file except in compliance with
+ * * the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing, software
+ * * distributed under the License is distributed on an "AS IS" BASIS,
+ * * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * * See the License for the specific language governing permissions and
+ * * limitations under the License.
+ *
+ */
Review Comment:
updated all license header
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1701094900
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0a5621c43699e3cd636c99ef6cc048788f04459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1702217716
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0a5621c43699e3cd636c99ef6cc048788f04459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573)
* 1640805e55e219b1c512bde9650849613c03e0b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598)
* ffc02724376dc67f1d5426fc1d95cbf1725d0261 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1703844611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19520",
"triggerID" : "ab46e341122b1fe70ebc85aa0b1a97204b2e7f89",
"triggerType" : "PUSH"
}, {
"hash" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19572",
"triggerID" : "bb592b7615a5b20676747abfd6883bfd107cd1aa",
"triggerType" : "PUSH"
}, {
"hash" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19573",
"triggerID" : "d0a5621c43699e3cd636c99ef6cc048788f04459",
"triggerType" : "PUSH"
}, {
"hash" : "1640805e55e219b1c512bde9650849613c03e0b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19598",
"triggerID" : "1640805e55e219b1c512bde9650849613c03e0b9",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603",
"triggerID" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"triggerType" : "PUSH"
}, {
"hash" : "ffc02724376dc67f1d5426fc1d95cbf1725d0261",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615",
"triggerID" : "1703549054",
"triggerType" : "MANUAL"
}, {
"hash" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "84d10c95a1f6b5395a85014031ac32267ac42fbc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffc02724376dc67f1d5426fc1d95cbf1725d0261 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19603) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19615)
* 84d10c95a1f6b5395a85014031ac32267ac42fbc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1315314496
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
The original code fetches the props either from cmd line or global options, what is the purpose to support custom props per-service then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9558:
URL: https://github.com/apache/hudi/pull/9558#discussion_r1316719006
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCleaner.java:
##########
@@ -64,6 +64,13 @@ public HoodieCleaner(Config cfg, JavaSparkContext jssc) {
LOG.info("Creating Cleaner with configs : " + props.toString());
}
+ public HoodieCleaner(Config cfg, TypedProperties props, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ this.props = props;
Review Comment:
> n multi-table services we also fetch the props either from cmd line or global options
The existing constructor already satisfy the needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9558: [HUDI-6481] Support run multi tables services in a single spark job
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9558:
URL: https://github.com/apache/hudi/pull/9558#issuecomment-1707700226
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org