You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/24 14:07:52 UTC
[GitHub] [hudi] zhangyue19921010 opened a new pull request, #5416: [HUDI-3963] [RFC-51] Improve Hoodie Writing Efficiency Using Lock-Free Message Queue
zhangyue19921010 opened a new pull request, #5416:
URL: https://github.com/apache/hudi/pull/5416
please read https://github.com/apache/hudi/pull/5415 for details
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271157459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 3e44fb2397bdbfa9c63fab47e62f312451b91900 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029)
* 672fdc6665cb250678c0768f55c8086a3cbfc388 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040)
* c0c22741ed123bd13cff21dec9fb7efdc70b99f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989702909
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+
+ waitForProducersFinished(pool);
+ queue.getInnerQueue().shutdown();
+ consumer.get().finish();
+
+ return consumer.get().getResult();
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ private void waitForProducersFinished(ExecutorCompletionService<Boolean> pool) throws InterruptedException, ExecutionException {
+ for (int i = 0; i < producers.size(); i++) {
+ pool.take().get();
+ }
+ }
+
+ /**
+ * Start only consumer.
+ */
+ private void setupConsumer() {
+ DisruptorMessageHandler<O, E> handler = new DisruptorMessageHandler<>(consumer.get());
+
+ Disruptor<HoodieDisruptorEvent<O>> innerQueue = queue.getInnerQueue();
+ innerQueue.handleEventsWith(handler);
+ innerQueue.start();
+ }
+
+ @Override
+ public boolean isRemaining() {
+ return !queue.isEmpty();
+ }
+
+ @Override
+ public void shutdownNow() {
+ producerExecutorService.shutdownNow();
+ consumerExecutorService.shutdownNow();
+ queue.close();
+ }
+
+ @Override
+ public DisruptorMessageQueue<I, O> getQueue() {
+ return queue;
+ }
+
+ @Override
+ public boolean awaitTermination() {
Review Comment:
> Please check my comment above: both BoundedInMemory and Disruptor Executors could share these sequences (for managing lifecycles of executor services)
changed!
> also, why return true directly. should we check for producerExecutorService termination and consumerExecutorService termination. similar logic as followed in BoundedInMemoryExecutor
Since we now have `HoodieExecutorBase` it holds common elements producerExecutorService, consumerExecutorService, producers and consumer and share the same logic for executor awaitTermination.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989936636
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorWaitStrategyType.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.keygen.constant.KeyGeneratorType;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public enum DisruptorWaitStrategyType {
+
+ /**
+ * The BlockingWaitStrategy is the slowest of the available wait strategies, but is the most conservative with the respect to CPU usage
+ * and will give the most consistent behaviour across the widest variety of deployment options.
+ */
+ BLOCKINGWAITSTRATEGY,
Review Comment:
changed!
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieConsumer<I, O> {
Review Comment:
Added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271466614
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 4587303118918c5e56ecb10732d9fcba43a90ee7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044)
* 4ba91d4ce8345b4917e1f402694a55d07bf2951c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989941353
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int producerNumber, Runnable preExecuteRunnable) {
Review Comment:
change to `totalProducers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271461268
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 4587303118918c5e56ecb10732d9fcba43a90ee7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044)
* 4ba91d4ce8345b4917e1f402694a55d07bf2951c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996574823
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent<O>> translator = new EventTranslator<HoodieDisruptorEvent<O>>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent<O> event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
+ return null;
+ }
+
+ @Override
+ public void close() {
+ queue.shutdown();
+ }
+
+ public void waitForConsumingFinished() throws InterruptedException {
+ while (!isEmpty()) {
+ Thread.sleep(1000);
Review Comment:
After refactoring, we don't need this `Consuming waiting` now. instead we will
1. Wait for all producers finished producing.
2. shutdown disruptor directly `call queue.close` which will waits until all events currently in the disruptor have been processed by all event processors
3. get and return result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280244923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* e230fbb946d8f33a4691129893e2559a3d8a1768 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211)
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 794e30be06f056e60824952e06c7eb7de4eb50c8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280247903
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* e230fbb946d8f33a4691129893e2559a3d8a1768 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211)
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 794e30be06f056e60824952e06c7eb7de4eb50c8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1173619958
Hi @leesf and @YuweiXiao Friendly ping. Would you mind to have a look at this PR at your convince?
Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #5416: [HUDI-3963] Improve Hoodie Writing Efficiency Using Lock-Free Message Queue
Posted by GitBox <gi...@apache.org>.
leesf commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1123851195
Thanks for the work, Can we file a RFC to describe the improvement in detail? @zhangyue19921010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280815374
Hi @alexeykudinkin All comments are addressed! PTAL
Thanks a again for your reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r997322284
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
Review Comment:
Finished!
Nice catch here!
Just unify the execute logic for both BIMQ and DMQ in HoodieExecutorBase#execute. There are 5 steps
setup()
startProducers()
startConsumer()
wait for result
finally postAction => do clean up works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009224931
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+
+/**
+ * Executor which orchestrates concurrent producers and consumers communicating through 'DisruptorMessageQueue'. This
+ * class takes as queue producer(s), consumer and transformer and exposes API to orchestrate
+ * concurrent execution of these actors communicating through disruptor
+ */
+public class DisruptorExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+ private final HoodieMessageQueue<I, O> queue;
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, Option.of(WaitStrategyFactory.DEFAULT_STRATEGY), Functions.noop());
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ super(producers, consumer, preExecuteRunnable);
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ @Override
+ public CompletableFuture<Void> startProducers() {
+ return CompletableFuture.allOf(producers.stream().map(producer -> {
+ return CompletableFuture.supplyAsync(() -> {
+ try {
+ producer.produce(queue);
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ }, producerExecutorService);
+ }).toArray(CompletableFuture[]::new));
+ }
+
+ @Override
+ protected void setup() {
+ ((DisruptorMessageQueue)queue).setHandlers(consumer.get());
+ ((DisruptorMessageQueue)queue).start();
+ }
+
+ @Override
+ protected void postAction() {
+ try {
+ queue.close();
+ super.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("Catch IOException while closing DisruptorMessageQueue", e);
+ }
+ }
+
+ @Override
+ protected CompletableFuture<E> startConsumer() {
+ return producerFuture.thenApplyAsync(res -> {
+ try {
+ queue.close();
Review Comment:
Changed! Only close the queue if this queue is not closed before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009214365
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1040,6 +1065,14 @@ public int getWriteBufferLimitBytes() {
return Integer.parseInt(getStringOrDefault(WRITE_BUFFER_LIMIT_BYTES_VALUE));
}
+ public Option<String> getWriteExecutorWaitStrategy() {
+ return Option.of(getString(WRITE_WAIT_STRATEGY));
+ }
+
+ public Option<Integer> getWriteBufferSize() {
Review Comment:
Sure, units to bytes and changed name to `getDisruptorWriteBufferSize()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1297293504
hi @nsivabalan Thanks a lot for your review. All comments are addressed. please take a look ：）
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552874
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(16);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ afterRecord.add((HoodieAvroRecord) record.record);
+ try {
+ IndexedRecord indexedRecord = (IndexedRecord)((HoodieAvroRecord) record.record)
+ .getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA).get();
+ afterIndexedRecord.add(indexedRecord);
+ } catch (IOException e) {
+ e.printStackTrace();
Review Comment:
All changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1282005627
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283292802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303)
* 9fe6bdae8829ea7a58d680cee1ea9c870d3549c8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1173619059
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989699375
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
+import java.util.concurrent.ThreadFactory;
+
+public class HoodieDaemonThreadFactory implements ThreadFactory {
+
+ private Runnable preExecuteRunnable;
+
+ public HoodieDaemonThreadFactory(Runnable preExecuteRunnable) {
+ this.preExecuteRunnable = preExecuteRunnable;
Review Comment:
```
public Runnable getPreExecuteRunnable() {
final TaskContext taskContext = TaskContext.get();
return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
}
```
We have to set up spark task context in each thread. So that we can get task info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989879896
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/FunctionBasedDisruptorProducer.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class FunctionBasedDisruptorProducer<I> extends DisruptorBasedProducer<I> {
Review Comment:
Removed this one and unify to `FunctionBasedQueueProducer` which produce records into `HoodieMessageQueue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1272296814
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 4ba91d4ce8345b4917e1f402694a55d07bf2951c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052)
* 447cb4510301af1c3ff1aebb3bd0a668872fc3f6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1162853327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1162848719
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1003968134
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/util/QueueBasedExecutorFactory.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.ExecutorType;
+import org.apache.hudi.common.util.queue.HoodieExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Iterator;
+import java.util.function.Function;
+
+public class QueueBasedExecutorFactory {
+
+ /**
+ * Create a new hoodie executor instance on demand.
+ */
+ public static HoodieExecutor create(HoodieWriteConfig hoodieConfig, Iterator inputItr, IteratorBasedQueueConsumer consumer,
Review Comment:
All changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1003973091
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutor.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.List;
+import java.util.concurrent.Future;
+
+/**
+ * HoodieExecutor which orchestrates concurrent producers and consumers communicating through a bounded in message queue.
+ */
+public interface HoodieExecutor<I, O, E> {
Review Comment:
Nice catch!
Just keep
```
E execute();
boolean isRemaining();
void shutdownNow();
boolean awaitTermination();
```
in `public interface HoodieExecutor<I, O, E> extends Closeable` which are used to control the lifecycle for hoodieExecutor for example `SparkLazyInsertIterable#computeNext`
And move all other function into `HoodieExecutorBase` and changed to protect except `startProducers` and `getQueue` which need to be call outside.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1003969271
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/WaitStrategyFactory.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import static org.apache.hudi.common.util.queue.DisruptorWaitStrategyType.BLOCKING_WAIT;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.BusySpinWaitStrategy;
+import com.lmax.disruptor.SleepingWaitStrategy;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.YieldingWaitStrategy;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+public class WaitStrategyFactory {
+
+ public static final String DEFAULT_STRATEGY = BLOCKING_WAIT.name();
+
+ /**
+ * Build WaitStrategy for disruptor
+ */
+ public static WaitStrategy build(Option<String> name) {
+ if (!name.isPresent()) {
+ return new BlockingWaitStrategy();
Review Comment:
Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1289947327
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524",
"triggerID" : "1288858105",
"triggerType" : "MANUAL"
}, {
"hash" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555",
"triggerID" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* e5c17f060235551dd9130e7bc7bbc33b294ebb18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524)
* d1970a3cfe95957775b7436d307107a8b1dee0e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1290758300
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524",
"triggerID" : "1288858105",
"triggerType" : "MANUAL"
}, {
"hash" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555",
"triggerID" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* d1970a3cfe95957775b7436d307107a8b1dee0e3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1147339221
> hi @zhangyue19921010 rebase this pr?
Sure. Rebased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1124854220
> Thanks for the work, Can we file a RFC to describe the improvement in detail? @zhangyue19921010
Thanks for your reminding @leesf.
Here is the claim RFC-53 PR https://github.com/apache/hudi/pull/5562
And here is the RFC-53 details https://github.com/apache/hudi/pull/5567
Please take a look :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r903357900
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/WaitStrategyFactory.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.BusySpinWaitStrategy;
+import com.lmax.disruptor.SleepingWaitStrategy;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.YieldingWaitStrategy;
+import org.apache.hudi.exception.HoodieException;
+
+public class WaitStrategyFactory {
+
+ public static final String DEFAULT_STRATEGY = "BlockingWaitStrategy";
+
+ /**
+ * Build WaitStrategy for disruptor
+ */
+ public static WaitStrategy build(String name) {
+ WaitStrategy waitStrategy = null;
+ if ("BlockingWaitStrategy".equals(name)) {
Review Comment:
Sure thing. Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1163024039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r903361063
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
Review Comment:
Actually, the execute logic is different between different executor, for example
Disruptor need to do `setup` works in `execute`. So I am not sure if we can unify all the execute method implementation :<
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1192451053
Hi @vinothchandar and @alexeykudinkin Sorry for the late response. I do several performance test on cluster
**Simple schema (several columns) + small amount of data** and **complex schema (dozens of columns) and large amount of data** using bulk_insert.
**Simple schema + small amount of data** like the bench marker showed, we may get better performance result from 60% to 2X
As for **complex schema (dozens of columns) and large amount of data**:
**1000 partitions, 100,000,000 records and dozens of columns**. We can get about 20% performance improved
Had to say that, the more data-producing faster than data consuming, the more benefit we can get from this PR. In this scenario, locking is the performance bottleneck.
For normal scenes, we can get about 20% performance improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552323
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
Review Comment:
all changed! Thanks.
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996574937
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent<O>> translator = new EventTranslator<HoodieDisruptorEvent<O>>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent<O> event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
+ return null;
+ }
+
+ @Override
+ public void close() {
+ queue.shutdown();
+ }
+
+ public void waitForConsumingFinished() throws InterruptedException {
+ while (!isEmpty()) {
+ Thread.sleep(1000);
Review Comment:
```
/**
* <p>Waits until all events currently in the disruptor have been processed by all event processors
* and then halts the processors. It is critical that publishing to the ring buffer has stopped
* before calling this method, otherwise it may never return.</p>
*
* <p>This method will not shutdown the executor, nor will it await the final termination of the
* processor threads.</p>
*/
public void shutdown()
{
try
{
shutdown(-1, TimeUnit.MILLISECONDS);
}
catch (final TimeoutException e)
{
exceptionHandler.handleOnShutdownException(e);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1282487519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281267923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* c748320fb229a7a013f4710ac1a2d68e60175d2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259)
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280635414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* d615f1f02d90ce8a26c3a9c7c0fd479e89de3155 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249)
* Unknown: [CANCELED](TBD)
* c748320fb229a7a013f4710ac1a2d68e60175d2e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] [RFC-51] Improve Hoodie Writing Efficiency Using Lock-Free Message Queue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1107856139
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 3778cf46256d3efb36646b29f06cf149e839045c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271721244
Hi @alexeykudinkin and @nsivabalan, Really appreciate for your review and comments! Also sorry for taking a few days to finish these comments.
Please take a look at your convince :)
If there are any omissions or other things that need to be modified, please feel free to let me know! Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989962337
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+
+ waitForProducersFinished(pool);
+ queue.getInnerQueue().shutdown();
+ consumer.get().finish();
+
+ return consumer.get().getResult();
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ private void waitForProducersFinished(ExecutorCompletionService<Boolean> pool) throws InterruptedException, ExecutionException {
+ for (int i = 0; i < producers.size(); i++) {
+ pool.take().get();
Review Comment:
Yeap. Just use this code to wait all the producer to complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1272290024
> Do we have tests for DistruptorProducers? I found tests only for DistruptorExecutor and DistruptorMessageQueue.
>
> Also, do we have tests to test out single producer and multiple producers? and can you summarize what kind of error cases have been tested. 1 producer thread crashing while others are still continuing to produce, memory too low to hold all produced records, but still no records should be dropped, etc.
Hi @nsivabalan Thanks a lot for reminding. Add more tests here including:
1. `TestBoundedInMemoryExecutorInSpark#testExecutor` ==> test common disruptor executor ingestion
2. `TestBoundedInMemoryExecutorInSpark#testInterruptExecutor` ==> test disruptor executor ingestion with interrupt
3. `TestDisruptorMessageQueue#testRecordReading` ==> test common single producer and single consumer reading
4. `TestDisruptorMessageQueue#testCompositeProducerRecordReading` ==> test multi-producers and single consumer
5. `TestDisruptorMessageQueue#testException` ==> test multi-producers which one producer thread crashing while others are still continuing to produce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1272324223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 447cb4510301af1c3ff1aebb3bd0a668872fc3f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989697092
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieMessageQueue.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieMessageQueue<I, O> {
Review Comment:
> Same comments as above (please ditto for all interfaces)
Changed all.
> This API seems incomplete: we do have methods to insert into the queue, but we don't have a method to take from the queue
Now there are 4 func in interface HoodieMessageQueue
```
public interface HoodieMessageQueue<I, O> {
/**
* Get the size of inner message queue.
*/
long size();
/**
* Insert a record into inner message queue.
*/
void insertRecord(I t) throws Exception;
/**
* Read records from inner message queue.
*/
Option<O> readNextRecord();
void close();
}
```
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/WaitStrategyFactory.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import static org.apache.hudi.common.util.queue.DisruptorWaitStrategyType.BLOCKINGWAITSTRATEGY;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.BusySpinWaitStrategy;
+import com.lmax.disruptor.SleepingWaitStrategy;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.YieldingWaitStrategy;
+import org.apache.hudi.exception.HoodieException;
+
+public class WaitStrategyFactory {
+
+ public static final String DEFAULT_STRATEGY = BLOCKINGWAITSTRATEGY.name();
+
+ /**
+ * Build WaitStrategy for disruptor
+ */
+ public static WaitStrategy build(String name) {
+
+ DisruptorWaitStrategyType strategyType = DisruptorWaitStrategyType.valueOf(name.toUpperCase());
+ switch (strategyType) {
+ case BLOCKINGWAITSTRATEGY:
Review Comment:
changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1294388711
>
> @zhangyue19921010 thank you very much for following up on all of the feedback!
>
> We're waiting on @nsivabalan to give us a green-light from his end, and then we should be good to go
Thanks a lot @alexeykudinkin and @nsivabalan !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1247917674
> Thanks for running the benchmarks @zhangyue19921010!
>
> Please sum them up in a table so that they are easier to digest and compare. I will prioritize reviewing this PR this week.
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009225575
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+
+/**
+ * Executor which orchestrates concurrent producers and consumers communicating through 'DisruptorMessageQueue'. This
+ * class takes as queue producer(s), consumer and transformer and exposes API to orchestrate
+ * concurrent execution of these actors communicating through disruptor
+ */
+public class DisruptorExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+ private final HoodieMessageQueue<I, O> queue;
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, Option.of(WaitStrategyFactory.DEFAULT_STRATEGY), Functions.noop());
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ super(producers, consumer, preExecuteRunnable);
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ @Override
+ public CompletableFuture<Void> startProducers() {
+ return CompletableFuture.allOf(producers.stream().map(producer -> {
+ return CompletableFuture.supplyAsync(() -> {
+ try {
+ producer.produce(queue);
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ }, producerExecutorService);
+ }).toArray(CompletableFuture[]::new));
+ }
+
+ @Override
+ protected void setup() {
+ ((DisruptorMessageQueue)queue).setHandlers(consumer.get());
+ ((DisruptorMessageQueue)queue).start();
+ }
+
+ @Override
+ protected void postAction() {
+ try {
+ queue.close();
Review Comment:
Nice catch. changed.
We align to shutdown the executor services and then do queue.close()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1296856174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524",
"triggerID" : "1288858105",
"triggerType" : "MANUAL"
}, {
"hash" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555",
"triggerID" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"triggerType" : "PUSH"
}, {
"hash" : "988e14a2e6ee6b2ac63573b530279ec37786041e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12682",
"triggerID" : "988e14a2e6ee6b2ac63573b530279ec37786041e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* d1970a3cfe95957775b7436d307107a8b1dee0e3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555)
* 988e14a2e6ee6b2ac63573b530279ec37786041e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12682)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1003969194
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutorBase.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HoodieExecutorBase holds common elements producerExecutorService, consumerExecutorService, producers and a single consumer.
+ * Also HoodieExecutorBase control the lifecycle of producerExecutorService and consumerExecutorService.
+ */
+public abstract class HoodieExecutorBase<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieExecutorBase.class);
+
+ private static final long TERMINATE_WAITING_TIME_SECS = 60L;
+ // Executor service used for launching write thread.
+ public final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ public final ExecutorService consumerExecutorService;
+ // Producers
+ public final List<HoodieProducer<I>> producers;
+ // Consumer
+ public final Option<IteratorBasedQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ public final Runnable preExecuteRunnable;
+
+ public List<Future<Boolean>> producerTasks;
+
+ public HoodieExecutorBase(List<HoodieProducer<I>> producers, Option<IteratorBasedQueueConsumer<O, E>> consumer,
+ Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new HoodieDaemonThreadFactory("producer", preExecuteRunnable));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ }
+
+ public boolean awaitTermination() {
+ // if current thread has been interrupted before awaitTermination was called, we still give
+ // executor a chance to proceeding. So clear the interrupt flag and reset it if needed before return.
+ boolean interruptedBefore = Thread.interrupted();
+ boolean producerTerminated = false;
+ boolean consumerTerminated = false;
+ try {
+ producerTerminated = producerExecutorService.awaitTermination(TERMINATE_WAITING_TIME_SECS, TimeUnit.SECONDS);
+ consumerTerminated = consumerExecutorService.awaitTermination(TERMINATE_WAITING_TIME_SECS, TimeUnit.SECONDS);
+ } catch (InterruptedException ie) {
+ // fail silently for any other interruption
+ }
+ // reset interrupt flag if needed
+ if (interruptedBefore) {
+ Thread.currentThread().interrupt();
+ }
+ return producerTerminated && consumerTerminated;
+ }
+
+ public void shutdownNow() {
+ producerExecutorService.shutdownNow();
+ consumerExecutorService.shutdownNow();
+ }
+
+ public void close() {
+ producerExecutorService.shutdown();
+ consumerExecutorService.shutdown();
+ }
+
+ /**
+ * Main API to run both production and consumption.
+ */
+ @Override
+ public E execute() {
+ try {
+ ValidationUtils.checkState(this.consumer.isPresent());
+ setup();
+ producerTasks = startProducers();
+ Future<E> future = startConsumer();
+ return future.get();
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
Review Comment:
This is follow the original code in BoundedInMemoryExecutor#execute.
Just keep it constant here.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/IteratorBasedHoodieMessageQueue.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.Iterator;
+
+/**
+ * IteratorBasedHoodieMessageQueue implements HoodieMessageQueue with Iterable
+ */
+public abstract class IteratorBasedHoodieMessageQueue<I, O> implements HoodieMessageQueue<I, O>, Iterable<O> {
Review Comment:
Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1289944050
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524",
"triggerID" : "1288858105",
"triggerType" : "MANUAL"
}, {
"hash" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* e5c17f060235551dd9130e7bc7bbc33b294ebb18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524)
* d1970a3cfe95957775b7436d307107a8b1dee0e3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] [RFC-51] Improve Hoodie Writing Efficiency Using Lock-Free Message Queue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1107856725
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 3778cf46256d3efb36646b29f06cf149e839045c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prasannarajaperumal commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
prasannarajaperumal commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1242207031
@zhangyue19921010 Like the abstraction and the pluggable queue implementation and circular buffer implementation. Looks good to me. @alexeykudinkin is the spark expert - once he ensures nothing else breaks, I think we are good to get this in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r984089070
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -230,6 +240,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.buffer.size")
+ .defaultValue(1024)
+ .withDocumentation("The size of the Disruptor Executor ring buffer, must be power of 2");
+
+ public static final ConfigProperty<String> WRITE_WAIT_STRATEGY = ConfigProperty
Review Comment:
Same as above
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutor.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.concurrent.ExecutorCompletionService;
+
+public abstract class HoodieExecutor<I, O, E> {
Review Comment:
- Let's convert this to an interface
- Please add a java-doc
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
+import java.util.concurrent.ThreadFactory;
+
+public class HoodieDaemonThreadFactory implements ThreadFactory {
+
+ private Runnable preExecuteRunnable;
+
+ public HoodieDaemonThreadFactory(Runnable preExecuteRunnable) {
+ this.preExecuteRunnable = preExecuteRunnable;
Review Comment:
Can you help me understand what kind of prologues we're planning to execute here?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
+import java.util.concurrent.ThreadFactory;
+
+public class HoodieDaemonThreadFactory implements ThreadFactory {
+
+ private Runnable preExecuteRunnable;
+
+ public HoodieDaemonThreadFactory(Runnable preExecuteRunnable) {
Review Comment:
If we're planning to have a custom factory it's a good idea to add custom name to the threads it produces (for them to be more easily identifiable)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
Review Comment:
Let's make it final
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -230,6 +240,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.buffer.size")
Review Comment:
Would suggest to sub-scope this to (for ex, `hoodie.write.executor.disruptor.buffer.size`)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorBasedProducer.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class DisruptorBasedProducer<I> implements HoodieProducer<I> {
Review Comment:
We should make both producers/consumers invariant of the Queue impl, in other words we should keep implementation details (BoundedLockingQueue, Disruptor, etc) abstracted away from them (there're simply no reason for them to be aware what's the implementation details of the queue).
- We should keep only `HoodieQueueProducer` (removing specific ones)
- Same for Consumers
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
Review Comment:
Please use `checkState` util for that
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieConsumer<I, O> {
+
+ /**
+ * Consumer One record.
+ */
+ protected abstract void consumeOneRecord(I record);
+
+ /**
+ * Notifies implementation that we have exhausted consuming records from queue.
+ */
+ protected abstract void finish();
+
+ /**
+ * Return result of consuming records so far.
+ */
+ protected abstract O getResult();
Review Comment:
Do we need this method in the interface?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/WaitStrategyFactory.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import static org.apache.hudi.common.util.queue.DisruptorWaitStrategyType.BLOCKINGWAITSTRATEGY;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.BusySpinWaitStrategy;
+import com.lmax.disruptor.SleepingWaitStrategy;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.YieldingWaitStrategy;
+import org.apache.hudi.exception.HoodieException;
+
+public class WaitStrategyFactory {
+
+ public static final String DEFAULT_STRATEGY = BLOCKINGWAITSTRATEGY.name();
+
+ /**
+ * Build WaitStrategy for disruptor
+ */
+ public static WaitStrategy build(String name) {
+
+ DisruptorWaitStrategyType strategyType = DisruptorWaitStrategyType.valueOf(name.toUpperCase());
+ switch (strategyType) {
+ case BLOCKINGWAITSTRATEGY:
Review Comment:
Let's use snake-case for these: `BLOCKING_WAIT_STRATEGY`
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieMessageQueue.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieMessageQueue<I, O> {
Review Comment:
Same comments as above (please ditto for all interfaces)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
Review Comment:
Please make sure you're using Javax annotations (not Jetbrains's ones)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEvent.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class HoodieDisruptorEvent<O> {
Review Comment:
What do we need to have this wrapper for?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
Review Comment:
Since Disruptor spawns its own threads, we can just stub this one out (no need to allocate any threads for it)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
Review Comment:
Let's create common base class `HoodieExecutorBase` abstracting common components for both executors (executor services, producers, consumers, etc)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+
+ waitForProducersFinished(pool);
+ queue.getInnerQueue().shutdown();
+ consumer.get().finish();
+
+ return consumer.get().getResult();
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ private void waitForProducersFinished(ExecutorCompletionService<Boolean> pool) throws InterruptedException, ExecutionException {
+ for (int i = 0; i < producers.size(); i++) {
+ pool.take().get();
+ }
+ }
+
+ /**
+ * Start only consumer.
+ */
+ private void setupConsumer() {
+ DisruptorMessageHandler<O, E> handler = new DisruptorMessageHandler<>(consumer.get());
+
+ Disruptor<HoodieDisruptorEvent<O>> innerQueue = queue.getInnerQueue();
+ innerQueue.handleEventsWith(handler);
+ innerQueue.start();
+ }
+
+ @Override
+ public boolean isRemaining() {
+ return !queue.isEmpty();
+ }
+
+ @Override
+ public void shutdownNow() {
+ producerExecutorService.shutdownNow();
+ consumerExecutorService.shutdownNow();
+ queue.close();
+ }
+
+ @Override
+ public DisruptorMessageQueue<I, O> getQueue() {
+ return queue;
+ }
+
+ @Override
+ public boolean awaitTermination() {
Review Comment:
Please check my comment above: both BoundedInMemory and Disruptor Executors could share these sequences (for managing lifecycles of executor services)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int producerNumber, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ if (producerNumber > 1) {
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, ProducerType.MULTI, waitStrategy);
+ } else {
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, ProducerType.SINGLE, waitStrategy);
+ }
+
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent<O>> translator = new EventTranslator<HoodieDisruptorEvent<O>>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent<O> event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public void close() {
+ queue.shutdown();
+ }
+
+ public Disruptor<HoodieDisruptorEvent<O>> getInnerQueue() {
Review Comment:
We should not be exposing this method
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutor.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.concurrent.ExecutorCompletionService;
+
+public abstract class HoodieExecutor<I, O, E> {
Review Comment:
Let's also add java-docs for the methods
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
Review Comment:
Strongly agree w/ @YuweiXiao here. Let's try our best to unify these as much as possible.
For the setup sequence we can create empty `setup` method in the base which will be no-op for Bounded
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieMessageQueue.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieMessageQueue<I, O> {
Review Comment:
This API seems incomplete: we do have methods to insert into the queue, but we don't have a method to take from the queue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271713313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 4ba91d4ce8345b4917e1f402694a55d07bf2951c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271160967
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 672fdc6665cb250678c0768f55c8086a3cbfc388 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040)
* c0c22741ed123bd13cff21dec9fb7efdc70b99f8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271079081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 3e44fb2397bdbfa9c63fab47e62f312451b91900 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029)
* 672fdc6665cb250678c0768f55c8086a3cbfc388 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989664909
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutor.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.concurrent.ExecutorCompletionService;
+
+public abstract class HoodieExecutor<I, O, E> {
Review Comment:
added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989940653
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEvent.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class HoodieDisruptorEvent<O> {
Review Comment:
added!
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEventFactory.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+
+public class HoodieDisruptorEventFactory<O> implements EventFactory<HoodieDisruptorEvent<O>> {
Review Comment:
added!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281381379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996551986
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -132,6 +134,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` "
+ "extract a key out of incoming records.");
+ public static final ConfigProperty<String> EXECUTOR_TYPE = ConfigProperty
+ .key("hoodie.write.executor.type")
+ .defaultValue("BOUNDED_IN_MEMORY")
Review Comment:
Sure, already created `enum ExecutorType`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1040,6 +1064,14 @@ public int getWriteBufferLimitBytes() {
return Integer.parseInt(getStringOrDefault(WRITE_BUFFER_LIMIT_BYTES_VALUE));
}
+ public String getWriteWaitStrategy() {
Review Comment:
Sure thing. Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996571604
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ this.hoodieExecutorBase = new HoodieExecutorBase<>(producers, consumer, preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(hoodieExecutorBase.getProducerExecutorService());
+ hoodieExecutorBase.getProducers().stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ hoodieExecutorBase.getPreExecuteRunnable().run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ ValidationUtils.checkState(hoodieExecutorBase.getConsumer().isPresent());
+ startConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+ return finishConsuming(pool);
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ @Override
+ public E finishConsuming(Object o) throws ExecutionException, InterruptedException {
Review Comment:
> @zhangyue19921010 let's make sure both BIMQ and Disruptor share execute method
Nice catch here!
Just unify the execute logic for both BIMQ and DMQ in `HoodieExecutorBase#execute`. There are 5 steps
1. setup()
2. startProducers()
3. startConsumer()
4. wait for result
5. finally postAction => do clean up works.
CC @YuweiXiao
> A few issue with this API: a) It's not type-safe (which we need to address), b) we don't need to pass in anything into it, instead this API simply
Should be called after all producers/consumers have finished
Should just be responsible for closing/cleaning up the executor's resources
Changed!
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ this.hoodieExecutorBase = new HoodieExecutorBase<>(producers, consumer, preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(hoodieExecutorBase.getProducerExecutorService());
+ hoodieExecutorBase.getProducers().stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ hoodieExecutorBase.getPreExecuteRunnable().run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ ValidationUtils.checkState(hoodieExecutorBase.getConsumer().isPresent());
+ startConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+ return finishConsuming(pool);
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ @Override
+ public E finishConsuming(Object o) throws ExecutionException, InterruptedException {
Review Comment:
> @zhangyue19921010 let's make sure both BIMQ and Disruptor share execute method
Nice catch here!
Just unify the execute logic for both BIMQ and DMQ in `HoodieExecutorBase#execute`. There are 5 steps
1. setup()
2. startProducers()
3. startConsumer()
4. wait for result
5. finally postAction => do clean up works.
CC @YuweiXiao
> A few issue with this API: a) It's not type-safe (which we need to address), b) we don't need to pass in anything into it, instead this API simply
Should be called after all producers/consumers have finished
Should just be responsible for closing/cleaning up the executor's resources
Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283008480
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283287432
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303)
* 9fe6bdae8829ea7a58d680cee1ea9c870d3549c8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283417403
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 9fe6bdae8829ea7a58d680cee1ea9c870d3549c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1284213374
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1162844161
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1184818569
> I am not sure why it may lead to a OOM when reader is reading too fast and writing is not able to keep up. Based on my limited knowledge, when consumers consume data quickly and producers produce data relatively slowly, consumers will continue to wait, which affects the throughput of the application. What we want is to allow the producer's data to be delivered to the consumer side as soon as possible, thereby improving CPU usage and throughput. This is why we want to use the queue of Disruptor to improve the efficiency of data flow through a lock-free design.
The scenario w/ OOM is the opposite of what you're describing: when producer is faster than the consumer -- in that case, w/o back-pressure mechanism producer will continue to produce filling in the in-memory buffers, with consumer not being able to catch up fast enough, which might eventually end up w/ OOM.
So i see some value in having the back-pressure mechanism but that frankly speaking doesn't require us to be grabbing locks in the hot-path -- it could be implemented even w/ Disruptor queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1182820038
@zhangyue19921010 This is very interesting work. Can we get some numbers on a cluster? - ideally writing to cloud storage or even HDFS.
Love to see the write performance improvement on a reasonable sized write. cc @alexeykudinkin @xushiyan as well as FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1162782261
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 8041a028f3f28b8d4745753b3ffffc452002b34e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102)
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1183666935
@zhangyue19921010 thanks for you contribution!
We indeed saw locks acquired by the queue in the current implementation of the `BoundedInMemoryExecutor` show up in our profiles, so this change is definitely a nice improvement!
That being said though, i think upside of using Disruptor is quite limited for Hudi due to our internal architecture: Disruptor's lock-free approach shines the brightest when you have _many_ Producers to _many_ consumers, in our case though Hudi in all cases but a few has exactly 1 producer and 1 consumer -- b/c we're using BIMQ only w/in a single Spark partition that is usually executed w/in a single Task on a single CPU core. In that sense BIMQ is just the way for us to balance b/w the speed of reading and the speed of writing -- if, say, reader is reading too fast and writing is not able to keep up, our memory buffers (filled by reader) might overflow leading to an OOM. BIMQ in that case will serve as a back-pressure mechanism slowing down the reader (with locks) until writer is able to catch up.
Avoiding locks in that path will be able to reduce our compute footprint by about ~10%, but from what i've seen so far i don't thin we'd be able to get more than that out of it unfortunately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1289470325
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524",
"triggerID" : "1288858105",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* e5c17f060235551dd9130e7bc7bbc33b294ebb18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1288872377
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524",
"triggerID" : "1288858105",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* e5c17f060235551dd9130e7bc7bbc33b294ebb18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1288858105
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1147352337
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 3778cf46256d3efb36646b29f06cf149e839045c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269)
* 8041a028f3f28b8d4745753b3ffffc452002b34e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1173667330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1177380885
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r876927787
##########
hudi-common/pom.xml:
##########
@@ -271,5 +271,12 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.lmax</groupId>
+ <artifactId>disruptor</artifactId>
+ <version>3.4.2</version>
Review Comment:
Is it better to define a version variable in project main pom? And put dependency declaration to main pom, too
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/BoundInMemoryExecutorBenchmark.scala:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.HoodieSparkUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.hudi.benchmark.{HoodieBenchmark, HoodieBenchmarkBase}
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{DataFrame, RowFactory, SaveMode, SparkSession}
+
+import scala.util.Random
+
+object BoundInMemoryExecutorBenchmark extends HoodieBenchmarkBase {
+
+ protected val spark: SparkSession = getSparkSession
+
+ val recordNumber = 1000000
+
+ def getSparkSession: SparkSession = SparkSession.builder()
+ .master("local[*]")
+ .appName(this.getClass.getCanonicalName)
+ .withExtensions(new HoodieSparkSessionExtension)
+ .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .config("spark.sql.session.timeZone", "CTT")
+ .config(sparkConf())
+ .getOrCreate()
+
+ def sparkConf(): SparkConf = {
+ val sparkConf = new SparkConf()
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ sparkConf.set("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ }
+ sparkConf
+ }
+
+ private def createDataFrame(number: Int): DataFrame = {
+ val schema = new StructType()
+ .add("c1", IntegerType)
+ .add("c2", StringType)
+
+ val rdd = spark.sparkContext.parallelize(0 to number, 2).map { item =>
+ val c1 = Integer.valueOf(item)
+ val c2 = s"abc"
+ RowFactory.create(c1, c2)
+ }
+ spark.createDataFrame(rdd, schema)
+ }
+
+ /**
+ * Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
+ * COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+ * ------------------------------------------------------------------------------------------------------------------------
+ * BoundInMemory Executor 5557 5607 70 0.2 5556.9 1.0X
+ * Disruptor Executor 2758 2778 28 0.4 2757.7 2.0X
+ */
+ private def cowTableDisruptorExecutorBenchmark(tableName: String = "executorBenchmark"): Unit = {
+ val df = createDataFrame(recordNumber)
Review Comment:
Could we put dataframe construction into the benchmark function, to avoid potential lazy construction of data frame and sharing of cached result, though benchmark results will also include the costs of source data construction.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/WaitStrategyFactory.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.BusySpinWaitStrategy;
+import com.lmax.disruptor.SleepingWaitStrategy;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.YieldingWaitStrategy;
+import org.apache.hudi.exception.HoodieException;
+
+public class WaitStrategyFactory {
+
+ public static final String DEFAULT_STRATEGY = "BlockingWaitStrategy";
+
+ /**
+ * Build WaitStrategy for disruptor
+ */
+ public static WaitStrategy build(String name) {
+ WaitStrategy waitStrategy = null;
+ if ("BlockingWaitStrategy".equals(name)) {
Review Comment:
Could we make these wait strategy as enum?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
Review Comment:
Could we unify the execute method implementation to base class for DisruptorExecutor & original bounded Executor? e.g., purely rely on consumer finish states.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/SparkLazyInsertIterable.java:
##########
@@ -77,24 +81,43 @@ public SparkLazyInsertIterable(Iterator<HoodieRecord<T>> recordItr,
@Override
protected List<WriteStatus> computeNext() {
// Executor service used for launching writer thread.
- BoundedInMemoryExecutor<HoodieRecord<T>, HoodieInsertValueGenResult<HoodieRecord>, List<WriteStatus>> bufferedIteratorExecutor =
- null;
+ HoodieExecutor<?, ?, List<WriteStatus>> executor = null;
try {
Schema schema = new Schema.Parser().parse(hoodieConfig.getSchema());
if (useWriterSchema) {
schema = HoodieAvroUtils.addMetadataFields(schema);
}
- bufferedIteratorExecutor =
- new BoundedInMemoryExecutor<>(hoodieConfig.getWriteBufferLimitBytes(), inputItr, getInsertHandler(),
+
+ String executorType = hoodieConfig.getExecutorType();
Review Comment:
We cloud put executorType -> executorTypeEnum conversion logic into the hooideConfig.getExecutorType(), to have more clear logic here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1166751036
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1163021664
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1012392921
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -18,117 +18,97 @@
package org.apache.hudi.common.util.queue;
-import org.apache.hudi.common.util.CustomizedThreadFactory;
import org.apache.hudi.common.util.DefaultSizeEstimator;
import org.apache.hudi.common.util.Functions;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.SizeEstimator;
import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
+import java.io.IOException;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutorCompletionService;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
import java.util.function.Function;
-import java.util.stream.Collectors;
/**
- * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory queue. This
+ * Executor which orchestrates concurrent producers and consumers communicating through 'BoundedInMemoryQueue'. This
* class takes as input the size limit, queue producer(s), consumer and transformer and exposes API to orchestrate
* concurrent execution of these actors communicating through a central bounded queue
*/
-public class BoundedInMemoryExecutor<I, O, E> {
+public class BoundedInMemoryExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
private static final Logger LOG = LogManager.getLogger(BoundedInMemoryExecutor.class);
- private static final long TERMINATE_WAITING_TIME_SECS = 60L;
- // Executor service used for launching write thread.
- private final ExecutorService producerExecutorService;
- // Executor service used for launching read thread.
- private final ExecutorService consumerExecutorService;
- // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
- private final BoundedInMemoryQueue<I, O> queue;
- // Producers
- private final List<BoundedInMemoryQueueProducer<I>> producers;
- // Consumer
- private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
- // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
- private final Runnable preExecuteRunnable;
+ private final HoodieMessageQueue<I, O> queue;
public BoundedInMemoryExecutor(final long bufferLimitInBytes, final Iterator<I> inputItr,
- BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
this(bufferLimitInBytes, producer, consumer, transformFunction, Functions.noop());
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, Collections.singletonList(producer), consumer, transformFunction, new DefaultSizeEstimator<>(), preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<BoundedInMemoryQueueProducer<I>> producers,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
final SizeEstimator<O> sizeEstimator, Runnable preExecuteRunnable) {
- this.producers = producers;
- this.consumer = consumer;
- this.preExecuteRunnable = preExecuteRunnable;
- // Ensure fixed thread for each producer thread
- this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
- // Ensure single thread for consumer
- this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
- this.queue = new BoundedInMemoryQueue<>(bufferLimitInBytes, transformFunction, sizeEstimator);
+ super(producers, consumer, preExecuteRunnable);
+ this.queue = new BoundedInMemoryQueueIterable<>(bufferLimitInBytes, transformFunction, sizeEstimator);
}
/**
- * Start all Producers.
+ * Start all producers at once.
*/
- public ExecutorCompletionService<Boolean> startProducers() {
+ @Override
+ public CompletableFuture<Void> startProducers() {
// Latch to control when and which producer thread will close the queue
final CountDownLatch latch = new CountDownLatch(producers.size());
- final ExecutorCompletionService<Boolean> completionService =
- new ExecutorCompletionService<Boolean>(producerExecutorService);
- producers.stream().map(producer -> {
- return completionService.submit(() -> {
+
+ return CompletableFuture.allOf(producers.stream().map(producer -> {
+ return CompletableFuture.supplyAsync(() -> {
try {
- preExecuteRunnable.run();
Review Comment:
gotcha.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280805987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* c748320fb229a7a013f4710ac1a2d68e60175d2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996554338
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(16);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ afterRecord.add((HoodieAvroRecord) record.record);
+ try {
+ IndexedRecord indexedRecord = (IndexedRecord)((HoodieAvroRecord) record.record)
+ .getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA).get();
+ afterIndexedRecord.add(indexedRecord);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(100, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+
+ assertEquals(beforeRecord, afterRecord);
+ assertEquals(beforeIndexedRecord, afterIndexedRecord);
+
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Test to ensure that we are reading all records from queue iterator when we have multiple producers.
+ */
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testCompositeProducerRecordReading() throws Exception {
+ final int numRecords = 1000;
+ final int numProducers = 40;
+ final List<List<HoodieRecord>> recs = new ArrayList<>();
+
+ final DisruptorMessageQueue<HoodieRecord, HoodieLazyInsertIterable.HoodieInsertValueGenResult> queue =
+ new DisruptorMessageQueue(1024, getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA),
+ "BLOCKING_WAIT", numProducers, new Runnable() {
+ @Override
+ public void run() {
+ // do nothing.
+ }
+ });
+
+ // Record Key to <Producer Index, Rec Index within a producer>
+ Map<String, Tuple2<Integer, Integer>> keyToProducerAndIndexMap = new HashMap<>();
+
+ for (int i = 0; i < numProducers; i++) {
+ List<HoodieRecord> pRecs = dataGen.generateInserts(instantTime, numRecords);
+ int j = 0;
+ for (HoodieRecord r : pRecs) {
+ assertFalse(keyToProducerAndIndexMap.containsKey(r.getRecordKey()));
+ keyToProducerAndIndexMap.put(r.getRecordKey(), new Tuple2<>(i, j));
+ j++;
+ }
+ recs.add(pRecs);
+ }
+
+ List<DisruptorPublisher> disruptorPublishers = new ArrayList<>();
+ for (int i = 0; i < recs.size(); i++) {
+ final List<HoodieRecord> r = recs.get(i);
+ // Alternate between pull and push based iterators
+ if (i % 2 == 0) {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new IteratorBasedQueueProducer<>(r.iterator()), queue);
+ disruptorPublishers.add(publisher);
+ } else {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new FunctionBasedQueueProducer<>((buf) -> {
+ Iterator<HoodieRecord> itr = r.iterator();
+ while (itr.hasNext()) {
+ try {
+ buf.insertRecord(itr.next());
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+ return true;
+ }), queue);
+ disruptorPublishers.add(publisher);
+ }
+ }
+
+ // Used to ensure that consumer sees the records generated by a single producer in FIFO order
+ Map<Integer, Integer> lastSeenMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> -1));
+ Map<Integer, Integer> countMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> 0));
+
+
+ // setup consumer and start disruptor
+ DisruptorMessageHandler<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> handler =
+ new DisruptorMessageHandler<>(new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> payload) {
+ // Read recs and ensure we have covered all producer recs.
+ final HoodieRecord rec = payload.record;
+ Tuple2<Integer, Integer> producerPos = keyToProducerAndIndexMap.get(rec.getRecordKey());
+ Integer lastSeenPos = lastSeenMap.get(producerPos._1());
+ countMap.put(producerPos._1(), countMap.get(producerPos._1()) + 1);
+ lastSeenMap.put(producerPos._1(), lastSeenPos + 1);
+ // Ensure we are seeing the next record generated
+ assertEquals(lastSeenPos + 1, producerPos._2().intValue());
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return 0;
+ }
+ });
+
+
+ queue.setHandlers(handler);
+ queue.start();
+
+
+ // start to produce records
+ final List<Future<Boolean>> futureList = disruptorPublishers.stream().map(disruptorPublisher -> {
+ return executorService.submit(() -> {
+ disruptorPublisher.startProduce();
+ return true;
+ });
+ }).collect(Collectors.toList());
+
+ // Close queue
+ Future<Boolean> closeFuture = executorService.submit(() -> {
Review Comment:
Changed. Just use` List<Future<Boolean>> futureList` to collect all the feature and wait for it finished directly :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281158615
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* c748320fb229a7a013f4710ac1a2d68e60175d2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996553050
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ @Test
+ public void testInterruptExecutor() {
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(1024);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ try {
+ while (true) {
+ Thread.sleep(1000);
Review Comment:
Changed.
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(16);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ afterRecord.add((HoodieAvroRecord) record.record);
+ try {
+ IndexedRecord indexedRecord = (IndexedRecord)((HoodieAvroRecord) record.record)
+ .getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA).get();
+ afterIndexedRecord.add(indexedRecord);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(100, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+
+ assertEquals(beforeRecord, afterRecord);
+ assertEquals(beforeIndexedRecord, afterIndexedRecord);
+
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Test to ensure that we are reading all records from queue iterator when we have multiple producers.
+ */
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testCompositeProducerRecordReading() throws Exception {
+ final int numRecords = 1000;
+ final int numProducers = 40;
+ final List<List<HoodieRecord>> recs = new ArrayList<>();
+
+ final DisruptorMessageQueue<HoodieRecord, HoodieLazyInsertIterable.HoodieInsertValueGenResult> queue =
+ new DisruptorMessageQueue(1024, getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA),
+ "BLOCKING_WAIT", numProducers, new Runnable() {
+ @Override
+ public void run() {
+ // do nothing.
+ }
+ });
+
+ // Record Key to <Producer Index, Rec Index within a producer>
+ Map<String, Tuple2<Integer, Integer>> keyToProducerAndIndexMap = new HashMap<>();
+
+ for (int i = 0; i < numProducers; i++) {
+ List<HoodieRecord> pRecs = dataGen.generateInserts(instantTime, numRecords);
+ int j = 0;
+ for (HoodieRecord r : pRecs) {
+ assertFalse(keyToProducerAndIndexMap.containsKey(r.getRecordKey()));
+ keyToProducerAndIndexMap.put(r.getRecordKey(), new Tuple2<>(i, j));
+ j++;
+ }
+ recs.add(pRecs);
+ }
+
+ List<DisruptorPublisher> disruptorPublishers = new ArrayList<>();
+ for (int i = 0; i < recs.size(); i++) {
+ final List<HoodieRecord> r = recs.get(i);
+ // Alternate between pull and push based iterators
+ if (i % 2 == 0) {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new IteratorBasedQueueProducer<>(r.iterator()), queue);
+ disruptorPublishers.add(publisher);
+ } else {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new FunctionBasedQueueProducer<>((buf) -> {
+ Iterator<HoodieRecord> itr = r.iterator();
+ while (itr.hasNext()) {
+ try {
+ buf.insertRecord(itr.next());
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+ return true;
+ }), queue);
+ disruptorPublishers.add(publisher);
+ }
+ }
+
+ // Used to ensure that consumer sees the records generated by a single producer in FIFO order
+ Map<Integer, Integer> lastSeenMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> -1));
+ Map<Integer, Integer> countMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> 0));
+
+
+ // setup consumer and start disruptor
+ DisruptorMessageHandler<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> handler =
+ new DisruptorMessageHandler<>(new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> payload) {
+ // Read recs and ensure we have covered all producer recs.
+ final HoodieRecord rec = payload.record;
+ Tuple2<Integer, Integer> producerPos = keyToProducerAndIndexMap.get(rec.getRecordKey());
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552715
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
Review Comment:
Yeap, use `final List<HoodieRecord> consumedRecords = new ArrayList<>();` instead to check value and order.
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ @Test
+ public void testInterruptExecutor() {
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(1024);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ try {
+ while (true) {
+ Thread.sleep(1000);
+ }
+ } catch (InterruptedException ie) {
+ return;
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return 0;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> executor = null;
+ AtomicReference<Exception> actualException = new AtomicReference<>();
+ try {
+ executor = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> finalExecutor = executor;
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996569594
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ this.hoodieExecutorBase = new HoodieExecutorBase<>(producers, consumer, preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(hoodieExecutorBase.getProducerExecutorService());
+ hoodieExecutorBase.getProducers().stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ hoodieExecutorBase.getPreExecuteRunnable().run();
Review Comment:
Changed.
Add this func as a pre-hook in `HoodieDaemonThreadFactory` which will executor firstly in each thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281747356
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1284261611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009227154
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent> ringBuffer;
+
+ public DisruptorMessageQueue(Option<Integer> bufferSize, Function<I, O> transformFunction, Option<String> waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(new HoodieDisruptorEventFactory(), bufferSize.get(), threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent> translator = new EventTranslator<HoodieDisruptorEvent>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
+ return null;
+ }
+
+ @Override
+ public void markAsFailed(Throwable e) {
+ // do nothing.
+ }
+
+ @Override
+ public void close() {
+ // Waits until all events currently in the disruptor have been processed by all event processors
+ queue.shutdown();
+ }
+
+ public boolean isEmpty() {
+ return ringBuffer.getBufferSize() == ringBuffer.remainingCapacity();
+ }
+
+ public void setHandlers(IteratorBasedQueueConsumer consumer) {
Review Comment:
Sure. Use java reflection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009228962
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieMessageQueue.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Option;
+import java.io.Closeable;
+
+/**
+ * HoodieMessageQueue holds an internal message queue, and control the behavior of
+ * 1. insert record into internal message queue.
+ * 2. get record from internal message queue.
+ * 3. close internal message queue.
+ */
+public interface HoodieMessageQueue<I, O> extends Closeable {
+
+ /**
+ * Get the size of inner message queue.
+ */
+ long size();
Review Comment:
Changed the java docs. it refers to `Returns the number of elements in this queue.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989700987
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
Review Comment:
nice idea. Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989698772
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEvent.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class HoodieDisruptorEvent<O> {
Review Comment:
Yeap. as nsivabalan said, we need to warp record/message as an event that can works in disruptor world.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989943308
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
Review Comment:
Yeap.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
Review Comment:
Sure thing. changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1295355705
will review by tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1284718195
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283473240
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 9fe6bdae8829ea7a58d680cee1ea9c870d3549c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319)
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1270249374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 67d8cd3c14c8539a4033c9e4a99b16eef9a03c25 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008)
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r903357345
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/BoundInMemoryExecutorBenchmark.scala:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.HoodieSparkUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.hudi.benchmark.{HoodieBenchmark, HoodieBenchmarkBase}
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{DataFrame, RowFactory, SaveMode, SparkSession}
+
+import scala.util.Random
+
+object BoundInMemoryExecutorBenchmark extends HoodieBenchmarkBase {
+
+ protected val spark: SparkSession = getSparkSession
+
+ val recordNumber = 1000000
+
+ def getSparkSession: SparkSession = SparkSession.builder()
+ .master("local[*]")
+ .appName(this.getClass.getCanonicalName)
+ .withExtensions(new HoodieSparkSessionExtension)
+ .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .config("spark.sql.session.timeZone", "CTT")
+ .config(sparkConf())
+ .getOrCreate()
+
+ def sparkConf(): SparkConf = {
+ val sparkConf = new SparkConf()
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ sparkConf.set("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ }
+ sparkConf
+ }
+
+ private def createDataFrame(number: Int): DataFrame = {
+ val schema = new StructType()
+ .add("c1", IntegerType)
+ .add("c2", StringType)
+
+ val rdd = spark.sparkContext.parallelize(0 to number, 2).map { item =>
+ val c1 = Integer.valueOf(item)
+ val c2 = s"abc"
+ RowFactory.create(c1, c2)
+ }
+ spark.createDataFrame(rdd, schema)
+ }
+
+ /**
+ * Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
+ * COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+ * ------------------------------------------------------------------------------------------------------------------------
+ * BoundInMemory Executor 5557 5607 70 0.2 5556.9 1.0X
+ * Disruptor Executor 2758 2778 28 0.4 2757.7 2.0X
+ */
+ private def cowTableDisruptorExecutorBenchmark(tableName: String = "executorBenchmark"): Unit = {
+ val df = createDataFrame(recordNumber)
Review Comment:
Yeap, Changed. Also i run this benchmark several times and here are the results
```
OpenJDK 64-Bit Server VM 1.8.0_161-b14 on Linux 3.10.0-693.21.1.el7.x86_64
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BoundInMemory Executor 5629 5765 192 0.2 5628.9 1.0X
Disruptor Executor 2772 2862 127 0.4 2772.2 2.0X
```
```
OpenJDK 64-Bit Server VM 1.8.0_161-b14 on Linux 3.10.0-693.21.1.el7.x86_64
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BoundInMemory Executor 5738 5862 175 0.2 5737.8 1.0X
Disruptor Executor 2742 2809 95 0.4 2742.0 2.1X
```
```
OpenJDK 64-Bit Server VM 1.8.0_161-b14 on Linux 3.10.0-693.21.1.el7.x86_64
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BoundInMemory Executor 5348 5482 189 0.2 5348.3 1.0X
Disruptor Executor 2719 2868 211 0.4 2718.9 2.0X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1162729638
Thanks for @YuweiXiao 's review. Comments are addressed. PTAL :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1147397151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 3778cf46256d3efb36646b29f06cf149e839045c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269)
* 8041a028f3f28b8d4745753b3ffffc452002b34e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271688530
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271863767
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 4ba91d4ce8345b4917e1f402694a55d07bf2951c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989961673
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
Review Comment:
Nice catch here. Actually in disruptor we need to
1. create and setup `DisruptorMessageHandler` into `Disruptor`
2. start disruptor and create event slots in ring buffer.
3. start to produce records into event slot.
So we take step 1 and 2 as start consumer which actually is `setup consumer`.
and take step as start producer.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
Review Comment:
Nice catch here. Actually in disruptor we need to
1. create and setup `DisruptorMessageHandler` into `Disruptor`
2. start disruptor and create event slots in ring buffer.
3. start to produce records into event slot.
So we take step 1 and 2 as start consumer which actually is `setup consumer`.
and take step 3 as start producer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989699543
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
+import java.util.concurrent.ThreadFactory;
+
+public class HoodieDaemonThreadFactory implements ThreadFactory {
+
+ private Runnable preExecuteRunnable;
+
+ public HoodieDaemonThreadFactory(Runnable preExecuteRunnable) {
Review Comment:
Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989940930
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieMessageQueue.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieMessageQueue<I, O> {
+
+ public abstract long size();
+
+ public abstract void insertRecord(I t) throws Exception;
+
+ public abstract void close();
Review Comment:
Sure thing. Changed.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieProducer.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public interface HoodieProducer<I> {
Review Comment:
Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989934496
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -129,6 +131,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` "
+ "extract a key out of incoming records.");
+ public static final ConfigProperty<String> EXECUTOR_TYPE = ConfigProperty
+ .key("hoodie.write.executor.type")
+ .defaultValue("BOUNDED_IN_MEMORY_EXECUTOR")
Review Comment:
Sure thing. Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1270364112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 67d8cd3c14c8539a4033c9e4a99b16eef9a03c25 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008)
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 3e44fb2397bdbfa9c63fab47e62f312451b91900 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552447
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ @Test
+ public void testInterruptExecutor() {
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(1024);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ try {
+ while (true) {
+ Thread.sleep(1000);
+ }
+ } catch (InterruptedException ie) {
+ return;
+ }
+ }
+
+ @Override
+ public void finish() {
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1278918191
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* e230fbb946d8f33a4691129893e2559a3d8a1768 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1278843853
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 447cb4510301af1c3ff1aebb3bd0a668872fc3f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069)
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* e230fbb946d8f33a4691129893e2559a3d8a1768 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280354130
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 794e30be06f056e60824952e06c7eb7de4eb50c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280438662
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 794e30be06f056e60824952e06c7eb7de4eb50c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241)
* 6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r997322504
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/BoundInMemoryExecutorBenchmark.scala:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.HoodieSparkUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.hudi.benchmark.{HoodieBenchmark, HoodieBenchmarkBase}
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{DataFrame, RowFactory, SaveMode, SparkSession}
+
+import scala.util.Random
+
+object BoundInMemoryExecutorBenchmark extends HoodieBenchmarkBase {
+
+ protected val spark: SparkSession = getSparkSession
+
+ val recordNumber = 1000000
+
+ def getSparkSession: SparkSession = SparkSession.builder()
+ .master("local[*]")
+ .appName(this.getClass.getCanonicalName)
+ .withExtensions(new HoodieSparkSessionExtension)
+ .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .config("spark.sql.session.timeZone", "CTT")
+ .config(sparkConf())
+ .getOrCreate()
+
+ def sparkConf(): SparkConf = {
+ val sparkConf = new SparkConf()
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ sparkConf.set("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ }
+ sparkConf
+ }
+
+ private def createDataFrame(number: Int): DataFrame = {
+ val schema = new StructType()
+ .add("c1", IntegerType)
+ .add("c2", StringType)
+
+ val rdd = spark.sparkContext.parallelize(0 to number, 2).map { item =>
+ val c1 = Integer.valueOf(item)
+ val c2 = s"abc"
+ RowFactory.create(c1, c2)
+ }
+ spark.createDataFrame(rdd, schema)
+ }
+
+ /**
+ * Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
+ * COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+ * ------------------------------------------------------------------------------------------------------------------------
+ * BoundInMemory Executor 5557 5607 70 0.2 5556.9 1.0X
+ * Disruptor Executor 2758 2778 28 0.4 2757.7 2.0X
+ */
+ private def cowTableDisruptorExecutorBenchmark(tableName: String = "executorBenchmark"): Unit = {
+ val df = createDataFrame(recordNumber)
Review Comment:
Not yet. Will have a try!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281750510
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996574030
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageHandler.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventHandler;
+
+public class DisruptorMessageHandler<O, E> implements EventHandler<HoodieDisruptorEvent<O>> {
Review Comment:
Follow the disruptor's usage framework. And need to be set into `queue.handleEventsWith(handler);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996574030
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageHandler.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventHandler;
+
+public class DisruptorMessageHandler<O, E> implements EventHandler<HoodieDisruptorEvent<O>> {
Review Comment:
Follow the disruptor's usage framework. And need to be set into `queue.handleEventsWith(handler);`
But we don't need a new class just change the `DisruptorMessageQueue#setHandlers` method and create an anonymous.
```
public void setHandlers(IteratorBasedQueueConsumer consumer) {
queue.handleEventsWith(new EventHandler<HoodieDisruptorEvent>() {
@Override
public void onEvent(HoodieDisruptorEvent event, long sequence, boolean endOfBatch) throws Exception {
consumer.consumeOneRecord(event.get());
event.clear();
}
});
}
```
Also removed `DisruptorMessageHandler` class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281255208
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* c748320fb229a7a013f4710ac1a2d68e60175d2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259)
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1166747210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1183668327
I unfortunately can't attach the screenshot from the recent profile to show where the 10% figure is coming from (GH crashes every time i try)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1177439169
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283640536
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996586884
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageHandler.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventHandler;
+
+public class DisruptorMessageHandler<O, E> implements EventHandler<HoodieDisruptorEvent<O>> {
+
+ private IteratorBasedQueueConsumer<O, E> consumer;
+
+ public DisruptorMessageHandler(IteratorBasedQueueConsumer<O, E> consumer) {
+ this.consumer = consumer;
+ }
+
+ @Override
+ public void onEvent(HoodieDisruptorEvent<O> event, long sequence, boolean endOfBatch) {
+ consumer.consumeOneRecord(event.get());
+ event.clear();
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552194
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/util/MessageQueueFactory.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.ExecutorType;
+import org.apache.hudi.common.util.queue.HoodieExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Iterator;
+import java.util.function.Function;
+
+public class MessageQueueFactory {
Review Comment:
Nice catch, Changed,
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestBoundedInMemoryExecutorInSpark.java:
##########
@@ -68,25 +68,26 @@ private Runnable getPreExecuteRunnable() {
return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
}
+ // common produce and consume based on disruptor executor
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552042
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1040,6 +1064,14 @@ public int getWriteBufferLimitBytes() {
return Integer.parseInt(getStringOrDefault(WRITE_BUFFER_LIMIT_BYTES_VALUE));
}
+ public String getWriteWaitStrategy() {
+ return getString(WRITE_WAIT_STRATEGY);
+ }
+
+ public int getWriteBufferSize() {
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996568602
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -46,68 +43,52 @@
* class takes as input the size limit, queue producer(s), consumer and transformer and exposes API to orchestrate
* concurrent execution of these actors communicating through a central bounded queue
*/
-public class BoundedInMemoryExecutor<I, O, E> {
+public class BoundedInMemoryExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
private static final Logger LOG = LogManager.getLogger(BoundedInMemoryExecutor.class);
- private static final long TERMINATE_WAITING_TIME_SECS = 60L;
- // Executor service used for launching write thread.
- private final ExecutorService producerExecutorService;
- // Executor service used for launching read thread.
- private final ExecutorService consumerExecutorService;
// Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
private final BoundedInMemoryQueue<I, O> queue;
- // Producers
- private final List<BoundedInMemoryQueueProducer<I>> producers;
- // Consumer
- private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
- // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
- private final Runnable preExecuteRunnable;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
Review Comment:
NICE CATCH HERE!
Changed. Also I didn't put queue into this `hoodieExecutorBase `
Because only a specific executor will have their corresponding queue. And some executor don't need this queue at all.
For example we will design a no-queue executor which could consume records from iterator directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996553966
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(16);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ afterRecord.add((HoodieAvroRecord) record.record);
+ try {
+ IndexedRecord indexedRecord = (IndexedRecord)((HoodieAvroRecord) record.record)
+ .getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA).get();
+ afterIndexedRecord.add(indexedRecord);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(100, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+
+ assertEquals(beforeRecord, afterRecord);
+ assertEquals(beforeIndexedRecord, afterIndexedRecord);
+
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Test to ensure that we are reading all records from queue iterator when we have multiple producers.
+ */
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testCompositeProducerRecordReading() throws Exception {
+ final int numRecords = 1000;
+ final int numProducers = 40;
+ final List<List<HoodieRecord>> recs = new ArrayList<>();
+
+ final DisruptorMessageQueue<HoodieRecord, HoodieLazyInsertIterable.HoodieInsertValueGenResult> queue =
+ new DisruptorMessageQueue(1024, getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA),
+ "BLOCKING_WAIT", numProducers, new Runnable() {
+ @Override
+ public void run() {
+ // do nothing.
+ }
+ });
+
+ // Record Key to <Producer Index, Rec Index within a producer>
+ Map<String, Tuple2<Integer, Integer>> keyToProducerAndIndexMap = new HashMap<>();
+
+ for (int i = 0; i < numProducers; i++) {
+ List<HoodieRecord> pRecs = dataGen.generateInserts(instantTime, numRecords);
+ int j = 0;
+ for (HoodieRecord r : pRecs) {
+ assertFalse(keyToProducerAndIndexMap.containsKey(r.getRecordKey()));
+ keyToProducerAndIndexMap.put(r.getRecordKey(), new Tuple2<>(i, j));
+ j++;
+ }
+ recs.add(pRecs);
+ }
+
+ List<DisruptorPublisher> disruptorPublishers = new ArrayList<>();
+ for (int i = 0; i < recs.size(); i++) {
+ final List<HoodieRecord> r = recs.get(i);
+ // Alternate between pull and push based iterators
+ if (i % 2 == 0) {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new IteratorBasedQueueProducer<>(r.iterator()), queue);
+ disruptorPublishers.add(publisher);
+ } else {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new FunctionBasedQueueProducer<>((buf) -> {
+ Iterator<HoodieRecord> itr = r.iterator();
+ while (itr.hasNext()) {
+ try {
+ buf.insertRecord(itr.next());
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+ return true;
+ }), queue);
+ disruptorPublishers.add(publisher);
+ }
+ }
+
+ // Used to ensure that consumer sees the records generated by a single producer in FIFO order
+ Map<Integer, Integer> lastSeenMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> -1));
+ Map<Integer, Integer> countMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> 0));
+
+
+ // setup consumer and start disruptor
+ DisruptorMessageHandler<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> handler =
+ new DisruptorMessageHandler<>(new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> payload) {
+ // Read recs and ensure we have covered all producer recs.
+ final HoodieRecord rec = payload.record;
+ Tuple2<Integer, Integer> producerPos = keyToProducerAndIndexMap.get(rec.getRecordKey());
+ Integer lastSeenPos = lastSeenMap.get(producerPos._1());
+ countMap.put(producerPos._1(), countMap.get(producerPos._1()) + 1);
+ lastSeenMap.put(producerPos._1(), lastSeenPos + 1);
+ // Ensure we are seeing the next record generated
+ assertEquals(lastSeenPos + 1, producerPos._2().intValue());
Review Comment:
We can only guarantee that the data of a single producer is in order, but the data sent by multiple producers to the queue is out of order. So that here use a map to collect and check the value and order for each producer.
Also this is follow the original design of `org.apache.hudi.execution#testCompositeProducerRecordReading`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280432697
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 794e30be06f056e60824952e06c7eb7de4eb50c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241)
* 6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009211037
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -233,6 +244,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.executor.disruptor.buffer.size")
+ .defaultValue(1024)
+ .withDocumentation("The size of the Disruptor Executor ring buffer, must be power of 2");
+
+ public static final ConfigProperty<String> WRITE_WAIT_STRATEGY = ConfigProperty
+ .key("hoodie.write.executor.disruptor.wait.strategy")
+ .defaultValue("BLOCKING_WAIT")
+ .withDocumentation("Strategy employed for making Disruptor Executor wait on a cursor.");
Review Comment:
Sure thing. Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009228332
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutorBase.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HoodieExecutorBase holds common elements producerExecutorService, consumerExecutorService, producers and a single consumer.
+ * Also HoodieExecutorBase control the lifecycle of producerExecutorService and consumerExecutorService.
+ */
+public abstract class HoodieExecutorBase<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieExecutorBase.class);
+
+ private static final long TERMINATE_WAITING_TIME_SECS = 60L;
+ // Executor service used for launching write thread.
+ public final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ public final ExecutorService consumerExecutorService;
+ // Producers
+ public final List<HoodieProducer<I>> producers;
+ // Consumer
+ public final Option<IteratorBasedQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ public final Runnable preExecuteRunnable;
+
+ public CompletableFuture<Void> producerFuture;
+
+ public HoodieExecutorBase(List<HoodieProducer<I>> producers, Option<IteratorBasedQueueConsumer<O, E>> consumer,
+ Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new HoodieDaemonThreadFactory("executor-queue-producer", preExecuteRunnable));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("executor-queue-consumer"));
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public abstract CompletableFuture<Void> startProducers();
+
+ /**
+ * Start consumer.
+ */
+ protected abstract CompletableFuture<E> startConsumer();
+
+ /**
+ * Closing/cleaning up the executor's resources after consuming finished.
+ */
+ protected abstract void postAction();
+
+ /**
+ * get bounded in message queue.
+ */
+ public abstract HoodieMessageQueue<I, O> getQueue();
+
+ /**
+ * set all the resources for current HoodieExecutor before start to produce and consume records.
+ */
+ protected abstract void setup();
+
+ public boolean awaitTermination() {
+ // if current thread has been interrupted before awaitTermination was called, we still give
+ // executor a chance to proceeding. So clear the interrupt flag and reset it if needed before return.
+ boolean interruptedBefore = Thread.interrupted();
+ boolean producerTerminated = false;
+ boolean consumerTerminated = false;
+ try {
+ producerTerminated = producerExecutorService.awaitTermination(TERMINATE_WAITING_TIME_SECS, TimeUnit.SECONDS);
+ consumerTerminated = consumerExecutorService.awaitTermination(TERMINATE_WAITING_TIME_SECS, TimeUnit.SECONDS);
+ } catch (InterruptedException ie) {
+ // fail silently for any other interruption
+ }
+ // reset interrupt flag if needed
+ if (interruptedBefore) {
+ Thread.currentThread().interrupt();
+ }
+ return producerTerminated && consumerTerminated;
+ }
+
+ /**
+ * Shutdown all the consumers and producers.
+ */
+ public void shutdownNow() {
Review Comment:
All changed. Just go through all the codes in this PR.
If there were anything missed, please let me know. Thanks！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009228962
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieMessageQueue.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Option;
+import java.io.Closeable;
+
+/**
+ * HoodieMessageQueue holds an internal message queue, and control the behavior of
+ * 1. insert record into internal message queue.
+ * 2. get record from internal message queue.
+ * 3. close internal message queue.
+ */
+public interface HoodieMessageQueue<I, O> extends Closeable {
+
+ /**
+ * Get the size of inner message queue.
+ */
+ long size();
Review Comment:
Changed. it refers to `Returns the number of elements in this queue.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009223711
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -18,117 +18,97 @@
package org.apache.hudi.common.util.queue;
-import org.apache.hudi.common.util.CustomizedThreadFactory;
import org.apache.hudi.common.util.DefaultSizeEstimator;
import org.apache.hudi.common.util.Functions;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.SizeEstimator;
import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
+import java.io.IOException;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutorCompletionService;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
import java.util.function.Function;
-import java.util.stream.Collectors;
/**
- * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory queue. This
+ * Executor which orchestrates concurrent producers and consumers communicating through 'BoundedInMemoryQueue'. This
* class takes as input the size limit, queue producer(s), consumer and transformer and exposes API to orchestrate
* concurrent execution of these actors communicating through a central bounded queue
*/
-public class BoundedInMemoryExecutor<I, O, E> {
+public class BoundedInMemoryExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
private static final Logger LOG = LogManager.getLogger(BoundedInMemoryExecutor.class);
- private static final long TERMINATE_WAITING_TIME_SECS = 60L;
- // Executor service used for launching write thread.
- private final ExecutorService producerExecutorService;
- // Executor service used for launching read thread.
- private final ExecutorService consumerExecutorService;
- // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
- private final BoundedInMemoryQueue<I, O> queue;
- // Producers
- private final List<BoundedInMemoryQueueProducer<I>> producers;
- // Consumer
- private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
- // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
- private final Runnable preExecuteRunnable;
+ private final HoodieMessageQueue<I, O> queue;
public BoundedInMemoryExecutor(final long bufferLimitInBytes, final Iterator<I> inputItr,
- BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
this(bufferLimitInBytes, producer, consumer, transformFunction, Functions.noop());
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, Collections.singletonList(producer), consumer, transformFunction, new DefaultSizeEstimator<>(), preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<BoundedInMemoryQueueProducer<I>> producers,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
final SizeEstimator<O> sizeEstimator, Runnable preExecuteRunnable) {
- this.producers = producers;
- this.consumer = consumer;
- this.preExecuteRunnable = preExecuteRunnable;
- // Ensure fixed thread for each producer thread
- this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
- // Ensure single thread for consumer
- this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
- this.queue = new BoundedInMemoryQueue<>(bufferLimitInBytes, transformFunction, sizeEstimator);
+ super(producers, consumer, preExecuteRunnable);
+ this.queue = new BoundedInMemoryQueueIterable<>(bufferLimitInBytes, transformFunction, sizeEstimator);
}
/**
- * Start all Producers.
+ * Start all producers at once.
*/
- public ExecutorCompletionService<Boolean> startProducers() {
+ @Override
+ public CompletableFuture<Void> startProducers() {
// Latch to control when and which producer thread will close the queue
final CountDownLatch latch = new CountDownLatch(producers.size());
- final ExecutorCompletionService<Boolean> completionService =
- new ExecutorCompletionService<Boolean>(producerExecutorService);
- producers.stream().map(producer -> {
- return completionService.submit(() -> {
+
+ return CompletableFuture.allOf(producers.stream().map(producer -> {
+ return CompletableFuture.supplyAsync(() -> {
try {
- preExecuteRunnable.run();
Review Comment:
Actually, we need to do preExecuteRunnable in Producer/Consumer thread of BoundedInMemory/DisruptorQueue
The original implementation is indeed a bit confusing, and it is simplified here.
expand the CustomizedThreadFactory and will do preExecuteRunnable in each thread if it wasn't null.
Also use this CustomizedThreadFactory to build producerExecutorService and consumerExecutorService
So that we don't need to call preExecuteRunnable manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #5416:
URL: https://github.com/apache/hudi/pull/5416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280642562
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* Unknown: [CANCELED](TBD)
* c748320fb229a7a013f4710ac1a2d68e60175d2e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1003968675
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+ final List<HoodieRecord> consumedRecords = new ArrayList<>();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(Option.of(8));
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ consumedRecords.add(record.record);
+ count++;
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), Option.of(WaitStrategyFactory.DEFAULT_STRATEGY), getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+
+ // collect all records and assert that consumed records are identical to produced ones
+ // assert there's no tampering, and that the ordering is preserved
+ assertEquals(hoodieRecords.size(), consumedRecords.size());
Review Comment:
changed!
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -18,96 +18,77 @@
package org.apache.hudi.common.util.queue;
-import org.apache.hudi.common.util.CustomizedThreadFactory;
import org.apache.hudi.common.util.DefaultSizeEstimator;
import org.apache.hudi.common.util.Functions;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.SizeEstimator;
import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
+import java.io.IOException;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorCompletionService;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import java.util.stream.Collectors;
/**
- * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory queue. This
+ * Executor which orchestrates concurrent producers and consumers communicating through 'BoundedInMemoryQueue'. This
* class takes as input the size limit, queue producer(s), consumer and transformer and exposes API to orchestrate
* concurrent execution of these actors communicating through a central bounded queue
*/
-public class BoundedInMemoryExecutor<I, O, E> {
+public class BoundedInMemoryExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
private static final Logger LOG = LogManager.getLogger(BoundedInMemoryExecutor.class);
- private static final long TERMINATE_WAITING_TIME_SECS = 60L;
- // Executor service used for launching write thread.
- private final ExecutorService producerExecutorService;
- // Executor service used for launching read thread.
- private final ExecutorService consumerExecutorService;
- // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
- private final BoundedInMemoryQueue<I, O> queue;
- // Producers
- private final List<BoundedInMemoryQueueProducer<I>> producers;
- // Consumer
- private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
- // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
- private final Runnable preExecuteRunnable;
+ private final HoodieMessageQueue<I, O> queue;
public BoundedInMemoryExecutor(final long bufferLimitInBytes, final Iterator<I> inputItr,
- BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
this(bufferLimitInBytes, producer, consumer, transformFunction, Functions.noop());
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, Collections.singletonList(producer), consumer, transformFunction, new DefaultSizeEstimator<>(), preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<BoundedInMemoryQueueProducer<I>> producers,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
final SizeEstimator<O> sizeEstimator, Runnable preExecuteRunnable) {
- this.producers = producers;
- this.consumer = consumer;
- this.preExecuteRunnable = preExecuteRunnable;
- // Ensure fixed thread for each producer thread
- this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
- // Ensure single thread for consumer
- this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ super(producers, consumer, preExecuteRunnable);
this.queue = new BoundedInMemoryQueue<>(bufferLimitInBytes, transformFunction, sizeEstimator);
}
/**
- * Start all Producers.
+ * Start all producers at once.
*/
- public ExecutorCompletionService<Boolean> startProducers() {
+ public List<Future<Boolean>> startProducers() {
Review Comment:
All changed.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutorBase.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HoodieExecutorBase holds common elements producerExecutorService, consumerExecutorService, producers and a single consumer.
+ * Also HoodieExecutorBase control the lifecycle of producerExecutorService and consumerExecutorService.
+ */
+public abstract class HoodieExecutorBase<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieExecutorBase.class);
+
+ private static final long TERMINATE_WAITING_TIME_SECS = 60L;
+ // Executor service used for launching write thread.
+ public final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ public final ExecutorService consumerExecutorService;
+ // Producers
+ public final List<HoodieProducer<I>> producers;
+ // Consumer
+ public final Option<IteratorBasedQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ public final Runnable preExecuteRunnable;
+
+ public List<Future<Boolean>> producerTasks;
+
+ public HoodieExecutorBase(List<HoodieProducer<I>> producers, Option<IteratorBasedQueueConsumer<O, E>> consumer,
+ Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new HoodieDaemonThreadFactory("producer", preExecuteRunnable));
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281793160
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281789564
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989940474
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieConsumer<I, O> {
+
+ /**
+ * Consumer One record.
+ */
+ protected abstract void consumeOneRecord(I record);
Review Comment:
Naming `consumeOneRecord` is followed the ori-naming from `BoundedInMemoryQueueConsumer`. But we now define `O consume(HoodieMessageQueue<?, I> queue);` instead of `consumeOneRecord` now.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
+import java.util.concurrent.ThreadFactory;
+
+public class HoodieDaemonThreadFactory implements ThreadFactory {
+
+ private Runnable preExecuteRunnable;
Review Comment:
change!
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
+import java.util.concurrent.ThreadFactory;
+
+public class HoodieDaemonThreadFactory implements ThreadFactory {
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989942523
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
Review Comment:
There's no use for this cnstr. Just follow the original definition in `BoundedInMemoryExecutor`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1272295990
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 4ba91d4ce8345b4917e1f402694a55d07bf2951c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052)
* 447cb4510301af1c3ff1aebb3bd0a668872fc3f6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1270546112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 3e44fb2397bdbfa9c63fab47e62f312451b91900 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1147651188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 8041a028f3f28b8d4745753b3ffffc452002b34e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1163028288
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1162845670
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1166739686
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989941632
##########
pom.xml:
##########
@@ -1069,6 +1070,12 @@
<version>${zk-curator.version}</version>
</dependency>
+ <dependency>
+ <groupId>com.lmax</groupId>
+ <artifactId>disruptor</artifactId>
+ <version>${disruptor.version}</version>
+ </dependency>
Review Comment:
no need in root. Just move it into `hudi-common`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r991641365
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2274,6 +2306,11 @@ public Builder withKeyGenerator(String keyGeneratorClass) {
return this;
}
+ public Builder withExecutorName(String executorClass) {
Review Comment:
Should this be `withExecutorType`?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
Review Comment:
Don't swallow exceptions, re-throw instead
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(16);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ afterRecord.add((HoodieAvroRecord) record.record);
+ try {
+ IndexedRecord indexedRecord = (IndexedRecord)((HoodieAvroRecord) record.record)
+ .getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA).get();
+ afterIndexedRecord.add(indexedRecord);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(100, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+
+ assertEquals(beforeRecord, afterRecord);
+ assertEquals(beforeIndexedRecord, afterIndexedRecord);
+
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Test to ensure that we are reading all records from queue iterator when we have multiple producers.
+ */
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testCompositeProducerRecordReading() throws Exception {
+ final int numRecords = 1000;
+ final int numProducers = 40;
+ final List<List<HoodieRecord>> recs = new ArrayList<>();
+
+ final DisruptorMessageQueue<HoodieRecord, HoodieLazyInsertIterable.HoodieInsertValueGenResult> queue =
+ new DisruptorMessageQueue(1024, getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA),
+ "BLOCKING_WAIT", numProducers, new Runnable() {
+ @Override
+ public void run() {
+ // do nothing.
+ }
+ });
+
+ // Record Key to <Producer Index, Rec Index within a producer>
+ Map<String, Tuple2<Integer, Integer>> keyToProducerAndIndexMap = new HashMap<>();
+
+ for (int i = 0; i < numProducers; i++) {
+ List<HoodieRecord> pRecs = dataGen.generateInserts(instantTime, numRecords);
+ int j = 0;
+ for (HoodieRecord r : pRecs) {
+ assertFalse(keyToProducerAndIndexMap.containsKey(r.getRecordKey()));
+ keyToProducerAndIndexMap.put(r.getRecordKey(), new Tuple2<>(i, j));
+ j++;
+ }
+ recs.add(pRecs);
+ }
+
+ List<DisruptorPublisher> disruptorPublishers = new ArrayList<>();
+ for (int i = 0; i < recs.size(); i++) {
+ final List<HoodieRecord> r = recs.get(i);
+ // Alternate between pull and push based iterators
+ if (i % 2 == 0) {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new IteratorBasedQueueProducer<>(r.iterator()), queue);
+ disruptorPublishers.add(publisher);
+ } else {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new FunctionBasedQueueProducer<>((buf) -> {
+ Iterator<HoodieRecord> itr = r.iterator();
+ while (itr.hasNext()) {
+ try {
+ buf.insertRecord(itr.next());
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+ return true;
+ }), queue);
+ disruptorPublishers.add(publisher);
+ }
+ }
+
+ // Used to ensure that consumer sees the records generated by a single producer in FIFO order
+ Map<Integer, Integer> lastSeenMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> -1));
+ Map<Integer, Integer> countMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> 0));
+
+
+ // setup consumer and start disruptor
+ DisruptorMessageHandler<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> handler =
+ new DisruptorMessageHandler<>(new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> payload) {
+ // Read recs and ensure we have covered all producer recs.
+ final HoodieRecord rec = payload.record;
+ Tuple2<Integer, Integer> producerPos = keyToProducerAndIndexMap.get(rec.getRecordKey());
+ Integer lastSeenPos = lastSeenMap.get(producerPos._1());
+ countMap.put(producerPos._1(), countMap.get(producerPos._1()) + 1);
+ lastSeenMap.put(producerPos._1(), lastSeenPos + 1);
+ // Ensure we are seeing the next record generated
+ assertEquals(lastSeenPos + 1, producerPos._2().intValue());
Review Comment:
Can we instead of asserting in a loop, collect all records per executor and then assert ordering in these lists?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -132,6 +134,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` "
+ "extract a key out of incoming records.");
+ public static final ConfigProperty<String> EXECUTOR_TYPE = ConfigProperty
+ .key("hoodie.write.executor.type")
+ .defaultValue("BOUNDED_IN_MEMORY")
Review Comment:
I believe @nsivabalan called this out previous, but don't see his comment anymore: let's create an enum with fixed values and use it here
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEventFactory.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+
+/**
+ * HoodieDisruptorEventFactory is used to create/preallocate HoodieDisruptorEvent.
+ *
+ */
+public class HoodieDisruptorEventFactory<O> implements EventFactory<HoodieDisruptorEvent<O>> {
Review Comment:
Same comment as above (let's make it impl detail of DisruptorQueue)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Access to a ThreadFactory instance.
+ * 1. All threads are created with setDaemon(true).
+ * 2. All threads execute preExecuteRunnable func once.
+ */
+public class HoodieDaemonThreadFactory implements ThreadFactory {
+
+ private final Runnable preExecuteRunnable;
+ private final AtomicInteger threadsNum = new AtomicInteger();
+ private final String namePattern;
+ private final String baseName = "Hoodie-disruptor-daemon-thread";
Review Comment:
Let's make template configurable (since ThreadFactory isn't coupled to Disruptor in any way)
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1040,6 +1064,14 @@ public int getWriteBufferLimitBytes() {
return Integer.parseInt(getStringOrDefault(WRITE_BUFFER_LIMIT_BYTES_VALUE));
}
+ public String getWriteWaitStrategy() {
Review Comment:
- nit: `getWriteExecutorWaitStrategy`
- Let's make it return `Option` (since config is optional)
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/util/MessageQueueFactory.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.ExecutorType;
+import org.apache.hudi.common.util.queue.HoodieExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Iterator;
+import java.util.function.Function;
+
+public class MessageQueueFactory {
Review Comment:
nit: Name is misleading here, let's ref executor in the name (like for ex `QueueBasedExecutorFactory`)
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestBoundedInMemoryExecutorInSpark.java:
##########
@@ -68,25 +68,26 @@ private Runnable getPreExecuteRunnable() {
return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
}
+ // common produce and consume based on disruptor executor
Review Comment:
Do we need this comment?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1040,6 +1064,14 @@ public int getWriteBufferLimitBytes() {
return Integer.parseInt(getStringOrDefault(WRITE_BUFFER_LIMIT_BYTES_VALUE));
}
+ public String getWriteWaitStrategy() {
+ return getString(WRITE_WAIT_STRATEGY);
+ }
+
+ public int getWriteBufferSize() {
Review Comment:
Similar comment to above one (this one will actually cause NPE, since we're returning primitive type)
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
Review Comment:
Let's actually instead collect all records and assert that consumed records are identical to produced ones (we want to assert there's no tampering, and that the ordering is preserved)
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
Review Comment:
Comment is out of date
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
Review Comment:
There's no need to override in that case (just adds boilerplate)
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ @Test
+ public void testInterruptExecutor() {
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(1024);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ try {
+ while (true) {
+ Thread.sleep(1000);
+ }
+ } catch (InterruptedException ie) {
+ return;
+ }
+ }
+
+ @Override
+ public void finish() {
Review Comment:
Same comment as above
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ @Test
+ public void testInterruptExecutor() {
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(1024);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ try {
+ while (true) {
+ Thread.sleep(1000);
+ }
+ } catch (InterruptedException ie) {
+ return;
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return 0;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> executor = null;
+ AtomicReference<Exception> actualException = new AtomicReference<>();
+ try {
+ executor = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> finalExecutor = executor;
Review Comment:
No need for another var, just drop `= null`
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(16);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ afterRecord.add((HoodieAvroRecord) record.record);
+ try {
+ IndexedRecord indexedRecord = (IndexedRecord)((HoodieAvroRecord) record.record)
+ .getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA).get();
+ afterIndexedRecord.add(indexedRecord);
+ } catch (IOException e) {
+ e.printStackTrace();
Review Comment:
Same comment as above, please ditto across the board
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ @Test
+ public void testInterruptExecutor() {
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(1024);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ try {
+ while (true) {
+ Thread.sleep(1000);
Review Comment:
Please avoid using sleeps in tests (it's a time-bomb). Instead just park the thread on a lock/latch/etc
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(16);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ afterRecord.add((HoodieAvroRecord) record.record);
+ try {
+ IndexedRecord indexedRecord = (IndexedRecord)((HoodieAvroRecord) record.record)
+ .getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA).get();
+ afterIndexedRecord.add(indexedRecord);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(100, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+
+ assertEquals(beforeRecord, afterRecord);
+ assertEquals(beforeIndexedRecord, afterIndexedRecord);
+
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Test to ensure that we are reading all records from queue iterator when we have multiple producers.
+ */
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testCompositeProducerRecordReading() throws Exception {
+ final int numRecords = 1000;
+ final int numProducers = 40;
+ final List<List<HoodieRecord>> recs = new ArrayList<>();
+
+ final DisruptorMessageQueue<HoodieRecord, HoodieLazyInsertIterable.HoodieInsertValueGenResult> queue =
+ new DisruptorMessageQueue(1024, getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA),
+ "BLOCKING_WAIT", numProducers, new Runnable() {
+ @Override
+ public void run() {
+ // do nothing.
+ }
+ });
+
+ // Record Key to <Producer Index, Rec Index within a producer>
+ Map<String, Tuple2<Integer, Integer>> keyToProducerAndIndexMap = new HashMap<>();
+
+ for (int i = 0; i < numProducers; i++) {
+ List<HoodieRecord> pRecs = dataGen.generateInserts(instantTime, numRecords);
+ int j = 0;
+ for (HoodieRecord r : pRecs) {
+ assertFalse(keyToProducerAndIndexMap.containsKey(r.getRecordKey()));
+ keyToProducerAndIndexMap.put(r.getRecordKey(), new Tuple2<>(i, j));
+ j++;
+ }
+ recs.add(pRecs);
+ }
+
+ List<DisruptorPublisher> disruptorPublishers = new ArrayList<>();
+ for (int i = 0; i < recs.size(); i++) {
+ final List<HoodieRecord> r = recs.get(i);
+ // Alternate between pull and push based iterators
+ if (i % 2 == 0) {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new IteratorBasedQueueProducer<>(r.iterator()), queue);
+ disruptorPublishers.add(publisher);
+ } else {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new FunctionBasedQueueProducer<>((buf) -> {
+ Iterator<HoodieRecord> itr = r.iterator();
+ while (itr.hasNext()) {
+ try {
+ buf.insertRecord(itr.next());
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+ return true;
+ }), queue);
+ disruptorPublishers.add(publisher);
+ }
+ }
+
+ // Used to ensure that consumer sees the records generated by a single producer in FIFO order
+ Map<Integer, Integer> lastSeenMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> -1));
+ Map<Integer, Integer> countMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> 0));
+
+
+ // setup consumer and start disruptor
+ DisruptorMessageHandler<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> handler =
+ new DisruptorMessageHandler<>(new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> payload) {
+ // Read recs and ensure we have covered all producer recs.
+ final HoodieRecord rec = payload.record;
+ Tuple2<Integer, Integer> producerPos = keyToProducerAndIndexMap.get(rec.getRecordKey());
Review Comment:
Let's use Hudi's Pair instead of Scala's Tuples
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ @Test
+ public void testInterruptExecutor() {
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(1024);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ try {
+ while (true) {
+ Thread.sleep(1000);
+ }
+ } catch (InterruptedException ie) {
+ return;
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return 0;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> executor = null;
+ AtomicReference<Exception> actualException = new AtomicReference<>();
+ try {
+ executor = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> finalExecutor = executor;
+
+ Future<?> future = pool.submit(() -> {
+ try {
+ finalExecutor.execute();
+ } catch (Exception e) {
+ actualException.set(e);
+ }
+
+ });
+ future.cancel(true);
+ future.get();
+ assertTrue(actualException.get() instanceof HoodieException);
Review Comment:
Instead of that please use `assertThrows` utility
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -46,68 +43,52 @@
* class takes as input the size limit, queue producer(s), consumer and transformer and exposes API to orchestrate
* concurrent execution of these actors communicating through a central bounded queue
*/
-public class BoundedInMemoryExecutor<I, O, E> {
+public class BoundedInMemoryExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
private static final Logger LOG = LogManager.getLogger(BoundedInMemoryExecutor.class);
- private static final long TERMINATE_WAITING_TIME_SECS = 60L;
- // Executor service used for launching write thread.
- private final ExecutorService producerExecutorService;
- // Executor service used for launching read thread.
- private final ExecutorService consumerExecutorService;
// Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
private final BoundedInMemoryQueue<I, O> queue;
- // Producers
- private final List<BoundedInMemoryQueueProducer<I>> producers;
- // Consumer
- private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
- // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
- private final Runnable preExecuteRunnable;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
Review Comment:
Why aggregating and not extending?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ this.hoodieExecutorBase = new HoodieExecutorBase<>(producers, consumer, preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(hoodieExecutorBase.getProducerExecutorService());
+ hoodieExecutorBase.getProducers().stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ hoodieExecutorBase.getPreExecuteRunnable().run();
Review Comment:
Let's embed this invocation into the executor itself (by creating a wrapper, so that there's no need to invoke it every time)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ this.hoodieExecutorBase = new HoodieExecutorBase<>(producers, consumer, preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(hoodieExecutorBase.getProducerExecutorService());
+ hoodieExecutorBase.getProducers().stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ hoodieExecutorBase.getPreExecuteRunnable().run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
Review Comment:
There's no need to do collect, let's do `forEach` instead
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(16);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ afterRecord.add((HoodieAvroRecord) record.record);
+ try {
+ IndexedRecord indexedRecord = (IndexedRecord)((HoodieAvroRecord) record.record)
+ .getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA).get();
+ afterIndexedRecord.add(indexedRecord);
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(100, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+
+ assertEquals(beforeRecord, afterRecord);
+ assertEquals(beforeIndexedRecord, afterIndexedRecord);
+
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ /**
+ * Test to ensure that we are reading all records from queue iterator when we have multiple producers.
+ */
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testCompositeProducerRecordReading() throws Exception {
+ final int numRecords = 1000;
+ final int numProducers = 40;
+ final List<List<HoodieRecord>> recs = new ArrayList<>();
+
+ final DisruptorMessageQueue<HoodieRecord, HoodieLazyInsertIterable.HoodieInsertValueGenResult> queue =
+ new DisruptorMessageQueue(1024, getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA),
+ "BLOCKING_WAIT", numProducers, new Runnable() {
+ @Override
+ public void run() {
+ // do nothing.
+ }
+ });
+
+ // Record Key to <Producer Index, Rec Index within a producer>
+ Map<String, Tuple2<Integer, Integer>> keyToProducerAndIndexMap = new HashMap<>();
+
+ for (int i = 0; i < numProducers; i++) {
+ List<HoodieRecord> pRecs = dataGen.generateInserts(instantTime, numRecords);
+ int j = 0;
+ for (HoodieRecord r : pRecs) {
+ assertFalse(keyToProducerAndIndexMap.containsKey(r.getRecordKey()));
+ keyToProducerAndIndexMap.put(r.getRecordKey(), new Tuple2<>(i, j));
+ j++;
+ }
+ recs.add(pRecs);
+ }
+
+ List<DisruptorPublisher> disruptorPublishers = new ArrayList<>();
+ for (int i = 0; i < recs.size(); i++) {
+ final List<HoodieRecord> r = recs.get(i);
+ // Alternate between pull and push based iterators
+ if (i % 2 == 0) {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new IteratorBasedQueueProducer<>(r.iterator()), queue);
+ disruptorPublishers.add(publisher);
+ } else {
+ DisruptorPublisher publisher = new DisruptorPublisher<>(new FunctionBasedQueueProducer<>((buf) -> {
+ Iterator<HoodieRecord> itr = r.iterator();
+ while (itr.hasNext()) {
+ try {
+ buf.insertRecord(itr.next());
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+ return true;
+ }), queue);
+ disruptorPublishers.add(publisher);
+ }
+ }
+
+ // Used to ensure that consumer sees the records generated by a single producer in FIFO order
+ Map<Integer, Integer> lastSeenMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> -1));
+ Map<Integer, Integer> countMap =
+ IntStream.range(0, numProducers).boxed().collect(Collectors.toMap(Function.identity(), x -> 0));
+
+
+ // setup consumer and start disruptor
+ DisruptorMessageHandler<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> handler =
+ new DisruptorMessageHandler<>(new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> payload) {
+ // Read recs and ensure we have covered all producer recs.
+ final HoodieRecord rec = payload.record;
+ Tuple2<Integer, Integer> producerPos = keyToProducerAndIndexMap.get(rec.getRecordKey());
+ Integer lastSeenPos = lastSeenMap.get(producerPos._1());
+ countMap.put(producerPos._1(), countMap.get(producerPos._1()) + 1);
+ lastSeenMap.put(producerPos._1(), lastSeenPos + 1);
+ // Ensure we are seeing the next record generated
+ assertEquals(lastSeenPos + 1, producerPos._2().intValue());
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return 0;
+ }
+ });
+
+
+ queue.setHandlers(handler);
+ queue.start();
+
+
+ // start to produce records
+ final List<Future<Boolean>> futureList = disruptorPublishers.stream().map(disruptorPublisher -> {
+ return executorService.submit(() -> {
+ disruptorPublisher.startProduce();
+ return true;
+ });
+ }).collect(Collectors.toList());
+
+ // Close queue
+ Future<Boolean> closeFuture = executorService.submit(() -> {
Review Comment:
nit: Instead we can use `ForkJoinPool` and join `CompletableFuture`s
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageHandler.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventHandler;
+
+public class DisruptorMessageHandler<O, E> implements EventHandler<HoodieDisruptorEvent<O>> {
+
+ private IteratorBasedQueueConsumer<O, E> consumer;
+
+ public DisruptorMessageHandler(IteratorBasedQueueConsumer<O, E> consumer) {
+ this.consumer = consumer;
+ }
+
+ @Override
+ public void onEvent(HoodieDisruptorEvent<O> event, long sequence, boolean endOfBatch) {
+ consumer.consumeOneRecord(event.get());
+ event.clear();
Review Comment:
Managing state of the Ring should be done w/in `DisruptorQueue` itself
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ this.hoodieExecutorBase = new HoodieExecutorBase<>(producers, consumer, preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(hoodieExecutorBase.getProducerExecutorService());
+ hoodieExecutorBase.getProducers().stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ hoodieExecutorBase.getPreExecuteRunnable().run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ ValidationUtils.checkState(hoodieExecutorBase.getConsumer().isPresent());
+ startConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+ return finishConsuming(pool);
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ @Override
+ public E finishConsuming(Object o) throws ExecutionException, InterruptedException {
Review Comment:
A few issue with this API: a) It's not type-safe (which we need to address), b) we don't need to pass in anything into it, instead this API simply
- Should be called after all producers/consumers have finished
- Should just be responsible for closing/cleaning up the executor's resources
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent<O>> translator = new EventTranslator<HoodieDisruptorEvent<O>>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent<O> event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
+ return null;
+ }
+
+ @Override
+ public void close() {
+ queue.shutdown();
+ }
+
+ public void waitForConsumingFinished() throws InterruptedException {
+ while (!isEmpty()) {
+ Thread.sleep(1000);
Review Comment:
We should not be doing busy-waiting (sleeping) in production code, instead we'd be waiting on a Future, until ones completes (there are `CompletableFuture`s specifically for that)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageHandler.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventHandler;
+
+public class DisruptorMessageHandler<O, E> implements EventHandler<HoodieDisruptorEvent<O>> {
Review Comment:
Why do we need Disruptor-specific handler?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorPublisher.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class DisruptorPublisher<I, O> {
Review Comment:
Same comment as above: why do we need Disruptor specific publisher?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
Review Comment:
@zhangyue19921010 let's make sure both BIMQ and Disruptor share `execute` method
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEvent.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class HoodieDisruptorEvent<O> {
Review Comment:
Alright, i see what this is now.
Let's make this a private subclass of the DisruptorQueue -- this is it's internal impl detail, and there's no reason to expose it outside of it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989704619
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
Review Comment:
changed！
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int producerNumber, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ if (producerNumber > 1) {
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, ProducerType.MULTI, waitStrategy);
+ } else {
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, ProducerType.SINGLE, waitStrategy);
+ }
+
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent<O>> translator = new EventTranslator<HoodieDisruptorEvent<O>>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent<O> event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public void close() {
+ queue.shutdown();
+ }
+
+ public Disruptor<HoodieDisruptorEvent<O>> getInnerQueue() {
Review Comment:
changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283429431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 9fe6bdae8829ea7a58d680cee1ea9c870d3549c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319)
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280769517
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281171191
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* c748320fb229a7a013f4710ac1a2d68e60175d2e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281684865
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552764
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicReference;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(8);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ count++;
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+ } finally {
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ }
+ }
+
+ @Test
+ public void testInterruptExecutor() {
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ExecutorService pool = Executors.newSingleThreadExecutor();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(1024);
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ try {
+ while (true) {
+ Thread.sleep(1000);
+ }
+ } catch (InterruptedException ie) {
+ return;
+ }
+ }
+
+ @Override
+ public void finish() {
+ }
+
+ @Override
+ protected Integer getResult() {
+ return 0;
+ }
+ };
+
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> executor = null;
+ AtomicReference<Exception> actualException = new AtomicReference<>();
+ try {
+ executor = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), WaitStrategyFactory.DEFAULT_STRATEGY, getPreExecuteRunnable());
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> finalExecutor = executor;
+
+ Future<?> future = pool.submit(() -> {
+ try {
+ finalExecutor.execute();
+ } catch (Exception e) {
+ actualException.set(e);
+ }
+
+ });
+ future.cancel(true);
+ future.get();
+ assertTrue(actualException.get() instanceof HoodieException);
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552828
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.DisruptorMessageHandler;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.DisruptorPublisher;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ e.printStackTrace();
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281699017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1288413070
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348)
* e5c17f060235551dd9130e7bc7bbc33b294ebb18 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1003973310
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent> ringBuffer;
+
+ public DisruptorMessageQueue(Option<Integer> bufferSize, Function<I, O> transformFunction, Option<String> waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(new HoodieDisruptorEventFactory(), bufferSize.get(), threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent> translator = new EventTranslator<HoodieDisruptorEvent>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
Review Comment:
All changed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1294014544
@zhangyue19921010 thank you very much for following up on all of the feedback!
We're waiting on @nsivabalan to give us a green-light from his end, and then we should be good to go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1293004257
> @zhangyue19921010 please rebase on the latest master. Test failures you observe have already been addressed on master.
Hi [alexeykudinkin](https://github.com/alexeykudinkin) Thanks a lot for reminding. rebase to master and all checks are green
Also comments are addressed. PTAL ：）
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1301505543
Great job buddy for patiently addressing all comments !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009227403
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/ExecutorType.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.keygen.constant.KeyGeneratorType;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Types of {@link org.apache.hudi.common.util.queue.HoodieExecutor}.
+ */
+public enum ExecutorType {
+
+ /**
+ * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory message queue using LinkedBlockingQueue.
+ */
+ BOUNDED_IN_MEMORY,
+
+ /**
+ * Executor which orchestrates concurrent producers and consumers communicating through disruptor as a lock free message queue
+ * to gain better writing performance. Although DisruptorExecutor is still an experimental feature.
+ */
+ DISRUPTOR;
+
+ public static List<String> getNames() {
+ List<String> names = new ArrayList<>(KeyGeneratorType.values().length);
Review Comment:
Sure, changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1297231532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524",
"triggerID" : "1288858105",
"triggerType" : "MANUAL"
}, {
"hash" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555",
"triggerID" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"triggerType" : "PUSH"
}, {
"hash" : "988e14a2e6ee6b2ac63573b530279ec37786041e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12682",
"triggerID" : "988e14a2e6ee6b2ac63573b530279ec37786041e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 988e14a2e6ee6b2ac63573b530279ec37786041e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12682)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1296849979
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12524",
"triggerID" : "1288858105",
"triggerType" : "MANUAL"
}, {
"hash" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555",
"triggerID" : "d1970a3cfe95957775b7436d307107a8b1dee0e3",
"triggerType" : "PUSH"
}, {
"hash" : "988e14a2e6ee6b2ac63573b530279ec37786041e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "988e14a2e6ee6b2ac63573b530279ec37786041e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* d1970a3cfe95957775b7436d307107a8b1dee0e3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12555)
* 988e14a2e6ee6b2ac63573b530279ec37786041e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009224221
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -138,61 +118,41 @@ private Future<E> startConsumer() {
} catch (Exception e) {
LOG.error("error consuming records", e);
queue.markAsFailed(e);
- throw e;
+ throw new HoodieException(e);
}
- });
+ }, consumerExecutorService);
}).orElse(CompletableFuture.completedFuture(null));
}
- /**
- * Main API to run both production and consumption.
- */
- public E execute() {
- try {
- startProducers();
- Future<E> future = startConsumer();
- // Wait for consumer to be done
- return future.get();
- } catch (InterruptedException ie) {
- shutdownNow();
- Thread.currentThread().interrupt();
- throw new HoodieException(ie);
- } catch (Exception e) {
- throw new HoodieException(e);
- }
+ @Override
+ public boolean isRemaining() {
+ return ((BoundedInMemoryQueueIterable)queue).iterator().hasNext();
Review Comment:
Sure changed.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+
+/**
+ * Executor which orchestrates concurrent producers and consumers communicating through 'DisruptorMessageQueue'. This
+ * class takes as queue producer(s), consumer and transformer and exposes API to orchestrate
+ * concurrent execution of these actors communicating through disruptor
+ */
+public class DisruptorExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+ private final HoodieMessageQueue<I, O> queue;
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1309800894
> Great job buddy for patiently addressing all comments !
Thanks a lot for your help! @alexeykudinkin and @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] [RFC-51] Improve Hoodie Writing Efficiency Using Lock-Free Message Queue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1107869989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 3778cf46256d3efb36646b29f06cf149e839045c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Improve Hoodie Writing Efficiency Using Lock-Free Message Queue
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1110378409
Hi @yihua @nsivabalan and @xushiyan Sorry to bother you. Do you have time to help review this PR?
Really really appreciate it. Thanks a lot :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1146978291
hi @zhangyue19921010 rebase this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1162729267
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 8041a028f3f28b8d4745753b3ffffc452002b34e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102)
* 1b868643f2f8794b4dd47f8e36124749df6effa3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283633132
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r998654598
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/util/QueueBasedExecutorFactory.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.util;
+
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.ExecutorType;
+import org.apache.hudi.common.util.queue.HoodieExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+
+import java.util.Iterator;
+import java.util.function.Function;
+
+public class QueueBasedExecutorFactory {
+
+ /**
+ * Create a new hoodie executor instance on demand.
+ */
+ public static HoodieExecutor create(HoodieWriteConfig hoodieConfig, Iterator inputItr, IteratorBasedQueueConsumer consumer,
Review Comment:
Let's make sure we're not dropping generics params
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent> ringBuffer;
+
+ public DisruptorMessageQueue(Option<Integer> bufferSize, Function<I, O> transformFunction, Option<String> waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(new HoodieDisruptorEventFactory(), bufferSize.get(), threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent> translator = new EventTranslator<HoodieDisruptorEvent>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
Review Comment:
We should throw `UnsupportedOperationException` in here
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ @Test
+ public void testExecutor() {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 128);
+ final List<HoodieRecord> consumedRecords = new ArrayList<>();
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(Option.of(8));
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
+ new IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer>() {
+
+ private int count = 0;
+
+ @Override
+ public void consumeOneRecord(HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord> record) {
+ consumedRecords.add(record.record);
+ count++;
+ }
+
+ @Override
+ protected Integer getResult() {
+ return count;
+ }
+ };
+ DisruptorExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+
+ try {
+ exec = new DisruptorExecutor(hoodieWriteConfig.getWriteBufferSize(), hoodieRecords.iterator(), consumer,
+ getTransformFunction(HoodieTestDataGenerator.AVRO_SCHEMA), Option.of(WaitStrategyFactory.DEFAULT_STRATEGY), getPreExecuteRunnable());
+ int result = exec.execute();
+ // It should buffer and write 100 records
+ assertEquals(128, result);
+ // There should be no remaining records in the buffer
+ assertFalse(exec.isRemaining());
+
+ // collect all records and assert that consumed records are identical to produced ones
+ // assert there's no tampering, and that the ordering is preserved
+ assertEquals(hoodieRecords.size(), consumedRecords.size());
Review Comment:
We can actually just do `assertEquals(oneList, anotherList)`
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/IteratorBasedHoodieMessageQueue.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.Iterator;
+
+/**
+ * IteratorBasedHoodieMessageQueue implements HoodieMessageQueue with Iterable
+ */
+public abstract class IteratorBasedHoodieMessageQueue<I, O> implements HoodieMessageQueue<I, O>, Iterable<O> {
Review Comment:
nit: It's rather `HoodieIterableMessageQueue`
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/WaitStrategyFactory.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import static org.apache.hudi.common.util.queue.DisruptorWaitStrategyType.BLOCKING_WAIT;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.BusySpinWaitStrategy;
+import com.lmax.disruptor.SleepingWaitStrategy;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.YieldingWaitStrategy;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+public class WaitStrategyFactory {
+
+ public static final String DEFAULT_STRATEGY = BLOCKING_WAIT.name();
+
+ /**
+ * Build WaitStrategy for disruptor
+ */
+ public static WaitStrategy build(Option<String> name) {
+ if (!name.isPresent()) {
+ return new BlockingWaitStrategy();
Review Comment:
nit: if name is not present we should fallback `strategyType` to default one
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutorBase.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HoodieExecutorBase holds common elements producerExecutorService, consumerExecutorService, producers and a single consumer.
+ * Also HoodieExecutorBase control the lifecycle of producerExecutorService and consumerExecutorService.
+ */
+public abstract class HoodieExecutorBase<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieExecutorBase.class);
+
+ private static final long TERMINATE_WAITING_TIME_SECS = 60L;
+ // Executor service used for launching write thread.
+ public final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ public final ExecutorService consumerExecutorService;
+ // Producers
+ public final List<HoodieProducer<I>> producers;
+ // Consumer
+ public final Option<IteratorBasedQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ public final Runnable preExecuteRunnable;
+
+ public List<Future<Boolean>> producerTasks;
+
+ public HoodieExecutorBase(List<HoodieProducer<I>> producers, Option<IteratorBasedQueueConsumer<O, E>> consumer,
+ Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new HoodieDaemonThreadFactory("producer", preExecuteRunnable));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ }
+
+ public boolean awaitTermination() {
+ // if current thread has been interrupted before awaitTermination was called, we still give
+ // executor a chance to proceeding. So clear the interrupt flag and reset it if needed before return.
+ boolean interruptedBefore = Thread.interrupted();
+ boolean producerTerminated = false;
+ boolean consumerTerminated = false;
+ try {
+ producerTerminated = producerExecutorService.awaitTermination(TERMINATE_WAITING_TIME_SECS, TimeUnit.SECONDS);
+ consumerTerminated = consumerExecutorService.awaitTermination(TERMINATE_WAITING_TIME_SECS, TimeUnit.SECONDS);
+ } catch (InterruptedException ie) {
+ // fail silently for any other interruption
+ }
+ // reset interrupt flag if needed
+ if (interruptedBefore) {
+ Thread.currentThread().interrupt();
+ }
+ return producerTerminated && consumerTerminated;
+ }
+
+ public void shutdownNow() {
+ producerExecutorService.shutdownNow();
+ consumerExecutorService.shutdownNow();
+ }
+
+ public void close() {
+ producerExecutorService.shutdown();
+ consumerExecutorService.shutdown();
+ }
+
+ /**
+ * Main API to run both production and consumption.
+ */
+ @Override
+ public E execute() {
+ try {
+ ValidationUtils.checkState(this.consumer.isPresent());
+ setup();
+ producerTasks = startProducers();
+ Future<E> future = startConsumer();
+ return future.get();
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
Review Comment:
Can you please elaborate with a comment on why we want to interrupt the current thread?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutorBase.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HoodieExecutorBase holds common elements producerExecutorService, consumerExecutorService, producers and a single consumer.
+ * Also HoodieExecutorBase control the lifecycle of producerExecutorService and consumerExecutorService.
+ */
+public abstract class HoodieExecutorBase<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieExecutorBase.class);
+
+ private static final long TERMINATE_WAITING_TIME_SECS = 60L;
+ // Executor service used for launching write thread.
+ public final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ public final ExecutorService consumerExecutorService;
+ // Producers
+ public final List<HoodieProducer<I>> producers;
+ // Consumer
+ public final Option<IteratorBasedQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ public final Runnable preExecuteRunnable;
+
+ public List<Future<Boolean>> producerTasks;
+
+ public HoodieExecutorBase(List<HoodieProducer<I>> producers, Option<IteratorBasedQueueConsumer<O, E>> consumer,
+ Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new HoodieDaemonThreadFactory("producer", preExecuteRunnable));
Review Comment:
nit: "executor-queue-producer", "executor-queue-consumer"
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -18,96 +18,77 @@
package org.apache.hudi.common.util.queue;
-import org.apache.hudi.common.util.CustomizedThreadFactory;
import org.apache.hudi.common.util.DefaultSizeEstimator;
import org.apache.hudi.common.util.Functions;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.SizeEstimator;
import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
+import java.io.IOException;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorCompletionService;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import java.util.stream.Collectors;
/**
- * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory queue. This
+ * Executor which orchestrates concurrent producers and consumers communicating through 'BoundedInMemoryQueue'. This
* class takes as input the size limit, queue producer(s), consumer and transformer and exposes API to orchestrate
* concurrent execution of these actors communicating through a central bounded queue
*/
-public class BoundedInMemoryExecutor<I, O, E> {
+public class BoundedInMemoryExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
private static final Logger LOG = LogManager.getLogger(BoundedInMemoryExecutor.class);
- private static final long TERMINATE_WAITING_TIME_SECS = 60L;
- // Executor service used for launching write thread.
- private final ExecutorService producerExecutorService;
- // Executor service used for launching read thread.
- private final ExecutorService consumerExecutorService;
- // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
- private final BoundedInMemoryQueue<I, O> queue;
- // Producers
- private final List<BoundedInMemoryQueueProducer<I>> producers;
- // Consumer
- private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
- // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
- private final Runnable preExecuteRunnable;
+ private final HoodieMessageQueue<I, O> queue;
public BoundedInMemoryExecutor(final long bufferLimitInBytes, final Iterator<I> inputItr,
- BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
this(bufferLimitInBytes, producer, consumer, transformFunction, Functions.noop());
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, Collections.singletonList(producer), consumer, transformFunction, new DefaultSizeEstimator<>(), preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<BoundedInMemoryQueueProducer<I>> producers,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
final SizeEstimator<O> sizeEstimator, Runnable preExecuteRunnable) {
- this.producers = producers;
- this.consumer = consumer;
- this.preExecuteRunnable = preExecuteRunnable;
- // Ensure fixed thread for each producer thread
- this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
- // Ensure single thread for consumer
- this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ super(producers, consumer, preExecuteRunnable);
this.queue = new BoundedInMemoryQueue<>(bufferLimitInBytes, transformFunction, sizeEstimator);
}
/**
- * Start all Producers.
+ * Start all producers at once.
*/
- public ExecutorCompletionService<Boolean> startProducers() {
+ public List<Future<Boolean>> startProducers() {
Review Comment:
Let's rebase all of our usage of `Future` onto `CompletableFuture` (which are much more capable and versatile than the former)
For that you just need to change how you trigger the execution doing:
```
CompletableFuture.runAsync(() -> ..., executorService);
```
Also, seems like this method is missing `@Override`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -132,6 +134,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` "
+ "extract a key out of incoming records.");
+ public static final ConfigProperty<String> EXECUTOR_TYPE = ConfigProperty
+ .key("hoodie.write.executor.type")
+ .defaultValue("BOUNDED_IN_MEMORY")
Review Comment:
Let's use it here as well
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent> ringBuffer;
+
+ public DisruptorMessageQueue(Option<Integer> bufferSize, Function<I, O> transformFunction, Option<String> waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(new HoodieDisruptorEventFactory(), bufferSize.get(), threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent> translator = new EventTranslator<HoodieDisruptorEvent>() {
Review Comment:
We can simplify this to
```
EventTranslator<HoodieDisruptorEvent> translator = (event, sequence) -> event.set(applied);
```
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent> ringBuffer;
+
+ public DisruptorMessageQueue(Option<Integer> bufferSize, Function<I, O> transformFunction, Option<String> waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(new HoodieDisruptorEventFactory(), bufferSize.get(), threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent> translator = new EventTranslator<HoodieDisruptorEvent>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
+ return null;
+ }
+
+ @Override
+ public void markAsFailed(Throwable e) {
+ // do nothing.
+ }
+
+ @Override
+ public void close() {
+ // Waits until all events currently in the disruptor have been processed by all event processors
+ queue.shutdown();
+ }
+
+ public boolean isEmpty() {
+ return ringBuffer.getBufferSize() == ringBuffer.remainingCapacity();
+ }
+
+ public void setHandlers(IteratorBasedQueueConsumer consumer) {
Review Comment:
Should this be package-private?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutor.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.List;
+import java.util.concurrent.Future;
+
+/**
+ * HoodieExecutor which orchestrates concurrent producers and consumers communicating through a bounded in message queue.
+ */
+public interface HoodieExecutor<I, O, E> {
Review Comment:
I'd suggest to reserve this interface for APIs only exposed to the actual user: in that case such only APIs are `execute` and `close`. All the other APIs are implementation detils (setup, startConumers, postAction, etc) and shouldn't be exposed outside actual implementations:
- Let's move all of these to a shared base class
- Let's keep them "protected"
Let's also make this extend `AutoCloseable`
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorPublisher.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class DisruptorPublisher<I, O> {
Review Comment:
@zhangyue19921010 forgot to remove this class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009218286
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ // ignore exception here.
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(Option.of(16));
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
Review Comment:
Nice Idea actually. We should keep the code as concise as possible.
https://issues.apache.org/jira/browse/HUDI-5106
Will working on it after this patch landed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1297914074
Also all the Azure jobs are passed now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1288748792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* e5c17f060235551dd9130e7bc7bbc33b294ebb18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281167180
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1281731671
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996574823
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent<O>> translator = new EventTranslator<HoodieDisruptorEvent<O>>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent<O> event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
+ return null;
+ }
+
+ @Override
+ public void close() {
+ queue.shutdown();
+ }
+
+ public void waitForConsumingFinished() throws InterruptedException {
+ while (!isEmpty()) {
+ Thread.sleep(1000);
Review Comment:
After refactoring, we don't need this wait now. instead we will
1. Wait for all producers finished producing.
2. shutdown disruptor directly `call queue.close` which will waits until all events currently in the disruptor have been processed by all event processors
3. get and return result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996570293
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ this.hoodieExecutorBase = new HoodieExecutorBase<>(producers, consumer, preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(hoodieExecutorBase.getProducerExecutorService());
+ hoodieExecutorBase.getProducers().stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ hoodieExecutorBase.getPreExecuteRunnable().run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
Review Comment:
just return these future list
```
public List<Future<Boolean>> startProducers() {
```
And for DisruptorExecutor#execute, it will wait for all these feature finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280203538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* e230fbb946d8f33a4691129893e2559a3d8a1768 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211)
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996571604
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.Future;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ private final HoodieExecutorBase<I, O, E> hoodieExecutorBase;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ this.hoodieExecutorBase = new HoodieExecutorBase<>(producers, consumer, preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(hoodieExecutorBase.getProducerExecutorService());
+ hoodieExecutorBase.getProducers().stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ hoodieExecutorBase.getPreExecuteRunnable().run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ ValidationUtils.checkState(hoodieExecutorBase.getConsumer().isPresent());
+ startConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+ return finishConsuming(pool);
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ @Override
+ public E finishConsuming(Object o) throws ExecutionException, InterruptedException {
Review Comment:
> @zhangyue19921010 let's make sure both BIMQ and Disruptor share execute method
Nice catch here!
Just unify the execute logic for both BIMQ and DMQ in `HoodieExecutorBase#execute`. There are 5 steps
1. setup()
2. startProducers()
3. startConsumer()
4. wait for result
5. finally postAction => do clean up works.
CC @YuweiXiao
> A few issue with this API: a) It's not type-safe (which we need to address), b) we don't need to pass in anything into it, instead this API simply
Should be called after all producers/consumers have finished
Should just be responsible for closing/cleaning up the executor's resources
Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1183759671
Hi @alexeykudinkin Thanks a lot for your attention! Also Glad to have more discussions :)
1. Disruptor not only performs well in the multi-production multi-consumption model, but also has good performance in single production and consumption scenarios due to the lock-free design. Based on https://github.com/LMAX-Exchange/disruptor/wiki/Performance-Results. And it is officially recommended to use a single-production model in the disruptor https://lmax-exchange.github.io/disruptor/user-guide/index.html#_introduction
> One of the best ways to improve performance in concurrent systems is to adhere to the [Single Writer Principle](https://mechanical-sympathy.blogspot.com/2011/09/single-writer-principle.html), this applies to the Disruptor. If you are in the situation where there will only ever be a single thread producing events into the Disruptor, then you can take advantage of this to gain additional performance.
2. I am not sure why it may lead to a OOM `when reader is reading too fast and writing is not able to keep up`. Based on my limited knowledge, when consumers consume data quickly and producers produce data relatively slowly, consumers will continue to wait, which affects the throughput of the application. What we want is to allow the producer's data to be delivered to the consumer side as soon as possible, thereby improving CPU usage and throughput. This is why we want to use the queue of Disruptor to improve the efficiency of data flow through a lock-free design.
3. I also fully agree with your point that the disruptor cannot solve all problems, nor can it achieve satisfactory optimization results in all scenarios. This depends on where the performance bottleneck of the user's hudi ingestion is. For example, like the scenario simulated in the benchmark, if the user's schema is relatively simple, the downstream consumption is fast, and the bottleneck is production or a lot of time is spent waiting for the data to be ready, then the Disruptor may be able to play the greatest value in this scenario.
As for conclusion `avoiding locks in that path will be able to reduce our compute footprint by about ~10%`, glad this will have a positive impact, at least it won't get worse :) Maybe we can do some tests on a variety of scenarios specially the bottleneck is the production data speed to see how the optimization works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1173671676
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1177446663
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 1b868643f2f8794b4dd47f8e36124749df6effa3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271311086
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* c0c22741ed123bd13cff21dec9fb7efdc70b99f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042)
* 4587303118918c5e56ecb10732d9fcba43a90ee7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r985104403
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -129,6 +131,14 @@ public class HoodieWriteConfig extends HoodieConfig {
.withDocumentation("Key generator class, that implements `org.apache.hudi.keygen.KeyGenerator` "
+ "extract a key out of incoming records.");
+ public static final ConfigProperty<String> EXECUTOR_TYPE = ConfigProperty
+ .key("hoodie.write.executor.type")
+ .defaultValue("BOUNDED_IN_MEMORY_EXECUTOR")
Review Comment:
can we introduce enum for this. also, we can remove "EXECUTOR" suffix from it. for eg, in case of Key gen, enum is named as KeyGeneratorType and value is just "SIMPLE". we don't call it as "SIMPOLE_KEY_GEN" as its repetitive.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEvent.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class HoodieDisruptorEvent<O> {
Review Comment:
java docs
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/BoundInMemoryExecutorBenchmark.scala:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.HoodieSparkUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.hudi.benchmark.{HoodieBenchmark, HoodieBenchmarkBase}
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{DataFrame, RowFactory, SaveMode, SparkSession}
+
+import scala.util.Random
+
+object BoundInMemoryExecutorBenchmark extends HoodieBenchmarkBase {
+
+ protected val spark: SparkSession = getSparkSession
+
+ val recordNumber = 1000000
+
+ def getSparkSession: SparkSession = SparkSession.builder()
+ .master("local[*]")
+ .appName(this.getClass.getCanonicalName)
+ .withExtensions(new HoodieSparkSessionExtension)
+ .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .config("spark.sql.session.timeZone", "CTT")
+ .config(sparkConf())
+ .getOrCreate()
+
+ def sparkConf(): SparkConf = {
+ val sparkConf = new SparkConf()
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ sparkConf.set("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ }
+ sparkConf
+ }
+
+ private def createDataFrame(number: Int): DataFrame = {
+ val schema = new StructType()
+ .add("c1", IntegerType)
+ .add("c2", StringType)
+
+ val rdd = spark.sparkContext.parallelize(0 to number, 2).map { item =>
+ val c1 = Integer.valueOf(item)
+ val c2 = s"abc"
+ RowFactory.create(c1, c2)
+ }
+ spark.createDataFrame(rdd, schema)
+ }
+
+ /**
+ * Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
+ * COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+ * ------------------------------------------------------------------------------------------------------------------------
+ * BoundInMemory Executor 5557 5607 70 0.2 5556.9 1.0X
+ * Disruptor Executor 2758 2778 28 0.4 2757.7 2.0X
+ */
+ private def cowTableDisruptorExecutorBenchmark(tableName: String = "executorBenchmark"): Unit = {
+ val df = createDataFrame(recordNumber)
Review Comment:
curious to know if you tried other Wait strategies
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+
+ waitForProducersFinished(pool);
+ queue.getInnerQueue().shutdown();
+ consumer.get().finish();
+
+ return consumer.get().getResult();
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ private void waitForProducersFinished(ExecutorCompletionService<Boolean> pool) throws InterruptedException, ExecutionException {
+ for (int i = 0; i < producers.size(); i++) {
+ pool.take().get();
+ }
+ }
+
+ /**
+ * Start only consumer.
+ */
+ private void setupConsumer() {
+ DisruptorMessageHandler<O, E> handler = new DisruptorMessageHandler<>(consumer.get());
+
+ Disruptor<HoodieDisruptorEvent<O>> innerQueue = queue.getInnerQueue();
+ innerQueue.handleEventsWith(handler);
+ innerQueue.start();
+ }
+
+ @Override
+ public boolean isRemaining() {
+ return !queue.isEmpty();
+ }
+
+ @Override
+ public void shutdownNow() {
+ producerExecutorService.shutdownNow();
+ consumerExecutorService.shutdownNow();
+ queue.close();
+ }
+
+ @Override
+ public DisruptorMessageQueue<I, O> getQueue() {
+ return queue;
+ }
+
+ @Override
+ public boolean awaitTermination() {
Review Comment:
also, why return true directly. should we check for producerExecutorService termination and consumerExecutorService termination. similar logic as followed in BoundedInMemoryExecutor
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieConsumer<I, O> {
Review Comment:
java docs please
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
+import java.util.concurrent.ThreadFactory;
+
+public class HoodieDaemonThreadFactory implements ThreadFactory {
+
+ private Runnable preExecuteRunnable;
Review Comment:
final
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieConsumer<I, O> {
+
+ /**
+ * Consumer One record.
+ */
+ protected abstract void consumeOneRecord(I record);
Review Comment:
`consumeRecord` should suffice IMO. why name it `consumeOneRecord`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -230,6 +240,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.buffer.size")
+ .defaultValue(1024)
+ .withDocumentation("The size of the Disruptor Executor ring buffer, must be power of 2");
+
+ public static final ConfigProperty<String> WRITE_WAIT_STRATEGY = ConfigProperty
+ .key("hoodie.write.wait.strategy")
+ .defaultValue("BlockingWaitStrategy")
Review Comment:
again, may be an enum for this too. and you can remove "Strategy" suffix.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/ExecutorType.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.keygen.constant.KeyGeneratorType;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Types of {@link org.apache.hudi.common.util.queue.HoodieExecutor}.
+ */
+public enum ExecutorType {
+
+ /**
+ * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory message queue using LinkedBlockingQueue.
+ */
+ BOUNDED_IN_MEMORY_EXECUTOR,
Review Comment:
oh, we do have an enum defined. may be we can remote "EXECUTOR" suffix. its repetitive
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieProducer.java:
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public interface HoodieProducer<I> {
Review Comment:
left comment elsewhere. we should move the produce method to this interface.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/SparkLazyInsertIterable.java:
##########
@@ -77,16 +80,28 @@ public SparkLazyInsertIterable(Iterator<HoodieRecord<T>> recordItr,
@Override
protected List<WriteStatus> computeNext() {
// Executor service used for launching writer thread.
- BoundedInMemoryExecutor<HoodieRecord<T>, HoodieInsertValueGenResult<HoodieRecord>, List<WriteStatus>> bufferedIteratorExecutor =
- null;
+ HoodieExecutor<?, ?, List<WriteStatus>> bufferedIteratorExecutor = null;
try {
Schema schema = new Schema.Parser().parse(hoodieConfig.getSchema());
if (useWriterSchema) {
schema = HoodieAvroUtils.addMetadataFields(schema);
}
- bufferedIteratorExecutor =
- new BoundedInMemoryExecutor<>(hoodieConfig.getWriteBufferLimitBytes(), inputItr, getInsertHandler(),
+
+ ExecutorType executorType = hoodieConfig.getExecutorType();
+
+ switch (executorType) {
+ case BOUNDED_IN_MEMORY_EXECUTOR:
Review Comment:
I have just started reviewing. not sure if we have similar code elsewhere. but can we move this to a util method so that code here is leaner and can be reused at other places.
```
bufferedIteratorExecutor = MessageQueueUtil/MessageQueueFactory.getMessageQueueExecutor(executorType size, itr, consumer, transformFn, preExecutorRunnable).
```
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorWaitStrategyType.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.keygen.constant.KeyGeneratorType;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public enum DisruptorWaitStrategyType {
+
+ /**
+ * The BlockingWaitStrategy is the slowest of the available wait strategies, but is the most conservative with the respect to CPU usage
+ * and will give the most consistent behaviour across the widest variety of deployment options.
+ */
+ BLOCKINGWAITSTRATEGY,
Review Comment:
can remove "STRATEGY" suffix. its repetitive
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int producerNumber, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ if (producerNumber > 1) {
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, ProducerType.MULTI, waitStrategy);
Review Comment:
we can do ternary operator.
```
this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, producerNumer >1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
```
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
Review Comment:
so, is this expected to be executed by every spark task i.e. for every producer?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
+import java.util.concurrent.ThreadFactory;
+
+public class HoodieDaemonThreadFactory implements ThreadFactory {
Review Comment:
java docs please
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieMessageQueue.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieMessageQueue<I, O> {
+
+ public abstract long size();
+
+ public abstract void insertRecord(I t) throws Exception;
+
+ public abstract void close();
Review Comment:
instead of adding this method explicitly, you can extend from Closeable interface.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/FunctionBasedDisruptorProducer.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class FunctionBasedDisruptorProducer<I> extends DisruptorBasedProducer<I> {
Review Comment:
can we add java docs to all new classes, interfaces, abstract classes please.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+
+ waitForProducersFinished(pool);
+ queue.getInnerQueue().shutdown();
Review Comment:
queue.close()
##########
pom.xml:
##########
@@ -1069,6 +1070,12 @@
<version>${zk-curator.version}</version>
</dependency>
+ <dependency>
+ <groupId>com.lmax</groupId>
+ <artifactId>disruptor</artifactId>
+ <version>${disruptor.version}</version>
+ </dependency>
Review Comment:
only hudi-common needs this right ? why add it to root pom ? can you help me understand.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int producerNumber, Runnable preExecuteRunnable) {
Review Comment:
`producerNumber` -> `producerCount` or `totalProducers`
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
Review Comment:
can we throw HoodieException here
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEventFactory.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+
+public class HoodieDisruptorEventFactory<O> implements EventFactory<HoodieDisruptorEvent<O>> {
Review Comment:
java docs
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/BoundInMemoryExecutorBenchmark.scala:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.HoodieSparkUtils
+import org.apache.spark.SparkConf
+import org.apache.spark.hudi.benchmark.{HoodieBenchmark, HoodieBenchmarkBase}
+import org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{DataFrame, RowFactory, SaveMode, SparkSession}
+
+import scala.util.Random
+
+object BoundInMemoryExecutorBenchmark extends HoodieBenchmarkBase {
+
+ protected val spark: SparkSession = getSparkSession
+
+ val recordNumber = 1000000
+
+ def getSparkSession: SparkSession = SparkSession.builder()
+ .master("local[*]")
+ .appName(this.getClass.getCanonicalName)
+ .withExtensions(new HoodieSparkSessionExtension)
+ .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+ .config("spark.sql.session.timeZone", "CTT")
+ .config(sparkConf())
+ .getOrCreate()
+
+ def sparkConf(): SparkConf = {
+ val sparkConf = new SparkConf()
+ if (HoodieSparkUtils.gteqSpark3_2) {
+ sparkConf.set("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
+ }
+ sparkConf
+ }
+
+ private def createDataFrame(number: Int): DataFrame = {
+ val schema = new StructType()
+ .add("c1", IntegerType)
+ .add("c2", StringType)
+
+ val rdd = spark.sparkContext.parallelize(0 to number, 2).map { item =>
+ val c1 = Integer.valueOf(item)
+ val c2 = s"abc"
+ RowFactory.create(c1, c2)
+ }
+ spark.createDataFrame(rdd, schema)
+ }
+
+ /**
+ * Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
+ * COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+ * ------------------------------------------------------------------------------------------------------------------------
+ * BoundInMemory Executor 5557 5607 70 0.2 5556.9 1.0X
+ * Disruptor Executor 2758 2778 28 0.4 2757.7 2.0X
+ */
+ private def cowTableDisruptorExecutorBenchmark(tableName: String = "executorBenchmark"): Unit = {
+ val df = createDataFrame(recordNumber)
Review Comment:
👏
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
Review Comment:
may I know what do we in this preExecute ? (as per this patch)
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
Review Comment:
In BoundedInMemoryExecutor, we start the producers and then start the consumer. do we know why was it done that way. Intuitive way is to start consumer and then stat producers.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEvent.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class HoodieDisruptorEvent<O> {
Review Comment:
guess thats how events are expected to be defined in disruptor framework. for eg https://lmax-exchange.github.io/disruptor/user-guide/index.html#_introduction check for LogEvent implementation here.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
Review Comment:
may I know where is this cnstr used?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/WaitStrategyFactory.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import static org.apache.hudi.common.util.queue.DisruptorWaitStrategyType.BLOCKINGWAITSTRATEGY;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.BusySpinWaitStrategy;
+import com.lmax.disruptor.SleepingWaitStrategy;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.YieldingWaitStrategy;
+import org.apache.hudi.exception.HoodieException;
+
+public class WaitStrategyFactory {
+
+ public static final String DEFAULT_STRATEGY = BLOCKINGWAITSTRATEGY.name();
+
+ /**
+ * Build WaitStrategy for disruptor
+ */
+ public static WaitStrategy build(String name) {
+
+ DisruptorWaitStrategyType strategyType = DisruptorWaitStrategyType.valueOf(name.toUpperCase());
+ switch (strategyType) {
+ case BLOCKINGWAITSTRATEGY:
Review Comment:
+1
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+
+ waitForProducersFinished(pool);
+ queue.getInnerQueue().shutdown();
+ consumer.get().finish();
+
+ return consumer.get().getResult();
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ private void waitForProducersFinished(ExecutorCompletionService<Boolean> pool) throws InterruptedException, ExecutionException {
+ for (int i = 0; i < producers.size(); i++) {
+ pool.take().get();
+ }
+ }
+
+ /**
+ * Start only consumer.
+ */
+ private void setupConsumer() {
+ DisruptorMessageHandler<O, E> handler = new DisruptorMessageHandler<>(consumer.get());
+
+ Disruptor<HoodieDisruptorEvent<O>> innerQueue = queue.getInnerQueue();
+ innerQueue.handleEventsWith(handler);
+ innerQueue.start();
+ }
+
+ @Override
+ public boolean isRemaining() {
+ return !queue.isEmpty();
+ }
+
+ @Override
+ public void shutdownNow() {
+ producerExecutorService.shutdownNow();
+ consumerExecutorService.shutdownNow();
+ queue.close();
+ }
+
+ @Override
+ public DisruptorMessageQueue<I, O> getQueue() {
+ return queue;
+ }
+
+ @Override
+ public boolean awaitTermination() {
Review Comment:
+1
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int producerNumber, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ if (producerNumber > 1) {
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, ProducerType.MULTI, waitStrategy);
+ } else {
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, ProducerType.SINGLE, waitStrategy);
+ }
+
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent<O>> translator = new EventTranslator<HoodieDisruptorEvent<O>>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent<O> event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public void close() {
+ queue.shutdown();
+ }
+
+ public Disruptor<HoodieDisruptorEvent<O>> getInnerQueue() {
Review Comment:
+1
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorBasedProducer.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class DisruptorBasedProducer<I> implements HoodieProducer<I> {
Review Comment:
+1
I see that we HoodieProducer is just an empty interface. and we have one abstract class for BoundedInMemory and another one for DistruptorBasedProducer. we should generify it and add produce() to base interface.
since HoodieMessageQueue interface is also shared among all producers and executor, we should definitely try to unify these.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+
+ waitForProducersFinished(pool);
+ queue.getInnerQueue().shutdown();
+ consumer.get().finish();
+
+ return consumer.get().getResult();
+ } catch (InterruptedException ie) {
+ shutdownNow();
+ Thread.currentThread().interrupt();
+ throw new HoodieException(ie);
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ }
+ }
+
+ private void waitForProducersFinished(ExecutorCompletionService<Boolean> pool) throws InterruptedException, ExecutionException {
+ for (int i = 0; i < producers.size(); i++) {
+ pool.take().get();
Review Comment:
sorry I don't understand this. In BoundedInMemoryExecutor, we have a count down latch and wait for all producers to countdown and then close the queue. can you help explain what are we doing here. I guess we want to wait for all producers to complete before we can call close/shutdown.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -230,6 +240,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.buffer.size")
+ .defaultValue(1024)
+ .withDocumentation("The size of the Disruptor Executor ring buffer, must be power of 2");
+
+ public static final ConfigProperty<String> WRITE_WAIT_STRATEGY = ConfigProperty
+ .key("hoodie.write.wait.strategy")
+ .defaultValue("BlockingWaitStrategy")
Review Comment:
not too strong on the suffix removal. feel free to take a call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989941128
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
+ try {
+ assert consumer.isPresent();
+ setupConsumer();
+ ExecutorCompletionService<Boolean> pool = startProducers();
+
+ waitForProducersFinished(pool);
+ queue.getInnerQueue().shutdown();
Review Comment:
changed!
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> extends HoodieMessageQueue<I, O> {
+
+ private final Disruptor<HoodieDisruptorEvent<O>> queue;
+ private final Function<I, O> transformFunction;
+ private RingBuffer<HoodieDisruptorEvent<O>> ringBuffer;
+
+ public DisruptorMessageQueue(int bufferSize, Function<I, O> transformFunction, String waitStrategyName, int producerNumber, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ if (producerNumber > 1) {
+ this.queue = new Disruptor<>(HoodieDisruptorEvent::new, bufferSize, threadFactory, ProducerType.MULTI, waitStrategy);
Review Comment:
chenged!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989936465
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/SparkLazyInsertIterable.java:
##########
@@ -77,16 +80,28 @@ public SparkLazyInsertIterable(Iterator<HoodieRecord<T>> recordItr,
@Override
protected List<WriteStatus> computeNext() {
// Executor service used for launching writer thread.
- BoundedInMemoryExecutor<HoodieRecord<T>, HoodieInsertValueGenResult<HoodieRecord>, List<WriteStatus>> bufferedIteratorExecutor =
- null;
+ HoodieExecutor<?, ?, List<WriteStatus>> bufferedIteratorExecutor = null;
try {
Schema schema = new Schema.Parser().parse(hoodieConfig.getSchema());
if (useWriterSchema) {
schema = HoodieAvroUtils.addMetadataFields(schema);
}
- bufferedIteratorExecutor =
- new BoundedInMemoryExecutor<>(hoodieConfig.getWriteBufferLimitBytes(), inputItr, getInsertHandler(),
+
+ ExecutorType executorType = hoodieConfig.getExecutorType();
+
+ switch (executorType) {
+ case BOUNDED_IN_MEMORY_EXECUTOR:
Review Comment:
Changed.
Currently only `parkLazyInsertIterable.java` will create this bufferedIteratorExecutor, but we really should move this to a util method.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/ExecutorType.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.keygen.constant.KeyGeneratorType;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Types of {@link org.apache.hudi.common.util.queue.HoodieExecutor}.
+ */
+public enum ExecutorType {
+
+ /**
+ * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory message queue using LinkedBlockingQueue.
+ */
+ BOUNDED_IN_MEMORY_EXECUTOR,
Review Comment:
changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271455933
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 4587303118918c5e56ecb10732d9fcba43a90ee7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989664466
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -230,6 +240,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.buffer.size")
Review Comment:
Changed.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -230,6 +240,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.buffer.size")
+ .defaultValue(1024)
+ .withDocumentation("The size of the Disruptor Executor ring buffer, must be power of 2");
+
+ public static final ConfigProperty<String> WRITE_WAIT_STRATEGY = ConfigProperty
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989927664
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
Review Comment:
```
@Override
public Runnable getPreExecuteRunnable() {
final TaskContext taskContext = TaskContext.get();
return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
}
```
defined in HoodieSparkTable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1278832114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 447cb4510301af1c3ff1aebb3bd0a668872fc3f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069)
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] [RFC-51] Improve Hoodie Writing Efficiency Using Lock-Free Message Queue
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1107850315
This Patch is tested on dev env, I think it's ready for review.
Although need more java docs here.
Here is the benchmark result between old BoundInMemoryExecutor and new DisruptorExecutor getting **2.0X** performance improved :)
```
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
COW Ingestion: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
BoundInMemory Executor 5557 5607 70 0.2 5556.9 1.0X
Disruptor Executor 2758 2778 28 0.4 2757.7 2.0X
```
Please read hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/BoundInMemoryExecutorBenchmark.scala for more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] [RFC-51] Improve Hoodie Writing Efficiency Using Lock-Free Message Queue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1107855600
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1230853505
Thanks for running the benchmarks @zhangyue19921010!
Please sum them up in a table so that they are easier to digest and compare. I will prioritize reviewing this PR this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r903353655
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/SparkLazyInsertIterable.java:
##########
@@ -77,24 +81,43 @@ public SparkLazyInsertIterable(Iterator<HoodieRecord<T>> recordItr,
@Override
protected List<WriteStatus> computeNext() {
// Executor service used for launching writer thread.
- BoundedInMemoryExecutor<HoodieRecord<T>, HoodieInsertValueGenResult<HoodieRecord>, List<WriteStatus>> bufferedIteratorExecutor =
- null;
+ HoodieExecutor<?, ?, List<WriteStatus>> executor = null;
try {
Schema schema = new Schema.Parser().parse(hoodieConfig.getSchema());
if (useWriterSchema) {
schema = HoodieAvroUtils.addMetadataFields(schema);
}
- bufferedIteratorExecutor =
- new BoundedInMemoryExecutor<>(hoodieConfig.getWriteBufferLimitBytes(), inputItr, getInsertHandler(),
+
+ String executorType = hoodieConfig.getExecutorType();
Review Comment:
Sure thing. Changed.
##########
hudi-common/pom.xml:
##########
@@ -271,5 +271,12 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.lmax</groupId>
+ <artifactId>disruptor</artifactId>
+ <version>3.4.2</version>
Review Comment:
Sure thing. Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1008719537
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -233,6 +244,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.executor.disruptor.buffer.size")
+ .defaultValue(1024)
+ .withDocumentation("The size of the Disruptor Executor ring buffer, must be power of 2");
+
+ public static final ConfigProperty<String> WRITE_WAIT_STRATEGY = ConfigProperty
+ .key("hoodie.write.executor.disruptor.wait.strategy")
+ .defaultValue("BLOCKING_WAIT")
+ .withDocumentation("Strategy employed for making Disruptor Executor wait on a cursor.");
Review Comment:
can we add other possible values in the documentation.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+
+/**
+ * Executor which orchestrates concurrent producers and consumers communicating through 'DisruptorMessageQueue'. This
+ * class takes as queue producer(s), consumer and transformer and exposes API to orchestrate
+ * concurrent execution of these actors communicating through disruptor
+ */
+public class DisruptorExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+ private final HoodieMessageQueue<I, O> queue;
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, Option.of(WaitStrategyFactory.DEFAULT_STRATEGY), Functions.noop());
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ super(producers, consumer, preExecuteRunnable);
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ @Override
+ public CompletableFuture<Void> startProducers() {
+ return CompletableFuture.allOf(producers.stream().map(producer -> {
+ return CompletableFuture.supplyAsync(() -> {
+ try {
+ producer.produce(queue);
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ }, producerExecutorService);
+ }).toArray(CompletableFuture[]::new));
+ }
+
+ @Override
+ protected void setup() {
+ ((DisruptorMessageQueue)queue).setHandlers(consumer.get());
+ ((DisruptorMessageQueue)queue).start();
+ }
+
+ @Override
+ protected void postAction() {
+ try {
+ queue.close();
Review Comment:
incase of shutdownNow, we shutdown the executorservices and then do queue. close.
where as here, we close queue first followed by shutting down executor services.
may I know whats the right approach. can we fix align.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -1040,6 +1065,14 @@ public int getWriteBufferLimitBytes() {
return Integer.parseInt(getStringOrDefault(WRITE_BUFFER_LIMIT_BYTES_VALUE));
}
+ public Option<String> getWriteExecutorWaitStrategy() {
+ return Option.of(getString(WRITE_WAIT_STRATEGY));
+ }
+
+ public Option<Integer> getWriteBufferSize() {
Review Comment:
this is confusing w/ already existing one (WRITE_BUFFER_LIMIT_BYTES_VALUE). Can we add units here. initially I thought this is also in bytes. but looks like its ring buffer size. We can also add "ring buffer" or "disruptor" to the name since this is applicable only to disruptor based queue.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -138,61 +118,41 @@ private Future<E> startConsumer() {
} catch (Exception e) {
LOG.error("error consuming records", e);
queue.markAsFailed(e);
- throw e;
+ throw new HoodieException(e);
}
- });
+ }, consumerExecutorService);
}).orElse(CompletableFuture.completedFuture(null));
}
- /**
- * Main API to run both production and consumption.
- */
- public E execute() {
- try {
- startProducers();
- Future<E> future = startConsumer();
- // Wait for consumer to be done
- return future.get();
- } catch (InterruptedException ie) {
- shutdownNow();
- Thread.currentThread().interrupt();
- throw new HoodieException(ie);
- } catch (Exception e) {
- throw new HoodieException(e);
- }
+ @Override
+ public boolean isRemaining() {
+ return ((BoundedInMemoryQueueIterable)queue).iterator().hasNext();
Review Comment:
minor. we could use getQueue() and avoid casting here.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+
+/**
+ * Executor which orchestrates concurrent producers and consumers communicating through 'DisruptorMessageQueue'. This
+ * class takes as queue producer(s), consumer and transformer and exposes API to orchestrate
+ * concurrent execution of these actors communicating through disruptor
+ */
+public class DisruptorExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+ private final HoodieMessageQueue<I, O> queue;
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, Option.of(WaitStrategyFactory.DEFAULT_STRATEGY), Functions.noop());
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ super(producers, consumer, preExecuteRunnable);
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ @Override
+ public CompletableFuture<Void> startProducers() {
+ return CompletableFuture.allOf(producers.stream().map(producer -> {
+ return CompletableFuture.supplyAsync(() -> {
+ try {
+ producer.produce(queue);
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw new HoodieException("Error producing records in disruptor executor", e);
+ }
+ return true;
+ }, producerExecutorService);
+ }).toArray(CompletableFuture[]::new));
+ }
+
+ @Override
+ protected void setup() {
+ ((DisruptorMessageQueue)queue).setHandlers(consumer.get());
+ ((DisruptorMessageQueue)queue).start();
+ }
+
+ @Override
+ protected void postAction() {
+ try {
+ queue.close();
+ super.close();
+ } catch (IOException e) {
+ throw new HoodieIOException("Catch IOException while closing DisruptorMessageQueue", e);
+ }
+ }
+
+ @Override
+ protected CompletableFuture<E> startConsumer() {
+ return producerFuture.thenApplyAsync(res -> {
+ try {
+ queue.close();
Review Comment:
I see we are calling queue.close() from many places. can you confirm that its idempotent and will not cause any exceptions.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/ExecutorType.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.keygen.constant.KeyGeneratorType;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * Types of {@link org.apache.hudi.common.util.queue.HoodieExecutor}.
+ */
+public enum ExecutorType {
+
+ /**
+ * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory message queue using LinkedBlockingQueue.
+ */
+ BOUNDED_IN_MEMORY,
+
+ /**
+ * Executor which orchestrates concurrent producers and consumers communicating through disruptor as a lock free message queue
+ * to gain better writing performance. Although DisruptorExecutor is still an experimental feature.
+ */
+ DISRUPTOR;
+
+ public static List<String> getNames() {
+ List<String> names = new ArrayList<>(KeyGeneratorType.values().length);
Review Comment:
isn't this ExecutorType.values() ?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutorBase.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HoodieExecutorBase holds common elements producerExecutorService, consumerExecutorService, producers and a single consumer.
+ * Also HoodieExecutorBase control the lifecycle of producerExecutorService and consumerExecutorService.
+ */
+public abstract class HoodieExecutorBase<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieExecutorBase.class);
+
+ private static final long TERMINATE_WAITING_TIME_SECS = 60L;
+ // Executor service used for launching write thread.
+ public final ExecutorService producerExecutorService;
Review Comment:
can we try to use protected wherever applicable. do we really need public for all these instance variables ?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorMessageQueue.java:
##########
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.execution;
+
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformFunction;
+
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hudi.common.model.HoodieAvroRecord;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.DisruptorMessageQueue;
+import org.apache.hudi.common.util.queue.FunctionBasedQueueProducer;
+import org.apache.hudi.common.util.queue.HoodieProducer;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueConsumer;
+import org.apache.hudi.common.util.queue.DisruptorExecutor;
+import org.apache.hudi.common.util.queue.IteratorBasedQueueProducer;
+import org.apache.hudi.common.util.queue.WaitStrategyFactory;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.testutils.HoodieClientTestHarness;
+import org.apache.spark.TaskContext;
+import org.apache.spark.TaskContext$;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import scala.Tuple2;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestDisruptorMessageQueue extends HoodieClientTestHarness {
+
+ private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initTestDataGenerator();
+ initExecutorServiceWithFixedThreadPool(2);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupResources();
+ }
+
+ private Runnable getPreExecuteRunnable() {
+ final TaskContext taskContext = TaskContext.get();
+ return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+ }
+
+ // Test to ensure that we are reading all records from queue iterator in the same order
+ // without any exceptions.
+ @SuppressWarnings("unchecked")
+ @Test
+ @Timeout(value = 60)
+ public void testRecordReading() throws Exception {
+
+ final List<HoodieRecord> hoodieRecords = dataGen.generateInserts(instantTime, 100);
+ ArrayList<HoodieRecord> beforeRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> beforeIndexedRecord = new ArrayList<>();
+ ArrayList<HoodieAvroRecord> afterRecord = new ArrayList<>();
+ ArrayList<IndexedRecord> afterIndexedRecord = new ArrayList<>();
+
+ hoodieRecords.forEach(record -> {
+ final HoodieAvroRecord originalRecord = (HoodieAvroRecord) record;
+ beforeRecord.add(originalRecord);
+ try {
+ final Option<IndexedRecord> originalInsertValue =
+ originalRecord.getData().getInsertValue(HoodieTestDataGenerator.AVRO_SCHEMA);
+ beforeIndexedRecord.add(originalInsertValue.get());
+ } catch (IOException e) {
+ // ignore exception here.
+ }
+ });
+
+ HoodieWriteConfig hoodieWriteConfig = mock(HoodieWriteConfig.class);
+ when(hoodieWriteConfig.getWriteBufferSize()).thenReturn(Option.of(16));
+ IteratorBasedQueueConsumer<HoodieLazyInsertIterable.HoodieInsertValueGenResult<HoodieRecord>, Integer> consumer =
Review Comment:
not required in this patch. But do you think we can have one set of tests and parametrize two diff queues. looking to reuse code. Bcoz, the way we test both bounded in memory queue and disruptor based one are same. just the initialization differs. but essentially its a functional testing of the queue.
let me know wdyt.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+
+/**
+ * Executor which orchestrates concurrent producers and consumers communicating through 'DisruptorMessageQueue'. This
+ * class takes as queue producer(s), consumer and transformer and exposes API to orchestrate
+ * concurrent execution of these actors communicating through disruptor
+ */
+public class DisruptorExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+ private final HoodieMessageQueue<I, O> queue;
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, final Iterator<I> inputItr,
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Option<String> waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final Option<Integer> bufferSize, HoodieProducer<I> producer,
Review Comment:
guess this constructor is not used. can we remove if not required
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieMessageQueue.java:
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.Option;
+import java.io.Closeable;
+
+/**
+ * HoodieMessageQueue holds an internal message queue, and control the behavior of
+ * 1. insert record into internal message queue.
+ * 2. get record from internal message queue.
+ * 3. close internal message queue.
+ */
+public interface HoodieMessageQueue<I, O> extends Closeable {
+
+ /**
+ * Get the size of inner message queue.
+ */
+ long size();
Review Comment:
minor. does this refer to total size, or pending size. not apparent from the name.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutorBase.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HoodieExecutorBase holds common elements producerExecutorService, consumerExecutorService, producers and a single consumer.
+ * Also HoodieExecutorBase control the lifecycle of producerExecutorService and consumerExecutorService.
+ */
+public abstract class HoodieExecutorBase<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieExecutorBase.class);
+
+ private static final long TERMINATE_WAITING_TIME_SECS = 60L;
+ // Executor service used for launching write thread.
+ public final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ public final ExecutorService consumerExecutorService;
+ // Producers
+ public final List<HoodieProducer<I>> producers;
+ // Consumer
+ public final Option<IteratorBasedQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ public final Runnable preExecuteRunnable;
+
+ public CompletableFuture<Void> producerFuture;
+
+ public HoodieExecutorBase(List<HoodieProducer<I>> producers, Option<IteratorBasedQueueConsumer<O, E>> consumer,
+ Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new HoodieDaemonThreadFactory("executor-queue-producer", preExecuteRunnable));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("executor-queue-consumer"));
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public abstract CompletableFuture<Void> startProducers();
+
+ /**
+ * Start consumer.
+ */
+ protected abstract CompletableFuture<E> startConsumer();
+
+ /**
+ * Closing/cleaning up the executor's resources after consuming finished.
+ */
+ protected abstract void postAction();
+
+ /**
+ * get bounded in message queue.
+ */
+ public abstract HoodieMessageQueue<I, O> getQueue();
+
+ /**
+ * set all the resources for current HoodieExecutor before start to produce and consume records.
+ */
+ protected abstract void setup();
+
+ public boolean awaitTermination() {
+ // if current thread has been interrupted before awaitTermination was called, we still give
+ // executor a chance to proceeding. So clear the interrupt flag and reset it if needed before return.
+ boolean interruptedBefore = Thread.interrupted();
+ boolean producerTerminated = false;
+ boolean consumerTerminated = false;
+ try {
+ producerTerminated = producerExecutorService.awaitTermination(TERMINATE_WAITING_TIME_SECS, TimeUnit.SECONDS);
+ consumerTerminated = consumerExecutorService.awaitTermination(TERMINATE_WAITING_TIME_SECS, TimeUnit.SECONDS);
+ } catch (InterruptedException ie) {
+ // fail silently for any other interruption
+ }
+ // reset interrupt flag if needed
+ if (interruptedBefore) {
+ Thread.currentThread().interrupt();
+ }
+ return producerTerminated && consumerTerminated;
+ }
+
+ /**
+ * Shutdown all the consumers and producers.
+ */
+ public void shutdownNow() {
Review Comment:
again, lets make protected or package private whereever applicbale.
and lets not make a method public just for testing purpose.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent> ringBuffer;
+
+ public DisruptorMessageQueue(Option<Integer> bufferSize, Function<I, O> transformFunction, Option<String> waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(new HoodieDisruptorEventFactory(), bufferSize.get(), threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent> translator = new EventTranslator<HoodieDisruptorEvent>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
+ return null;
+ }
+
+ @Override
+ public void markAsFailed(Throwable e) {
+ // do nothing.
+ }
+
+ @Override
+ public void close() {
+ // Waits until all events currently in the disruptor have been processed by all event processors
+ queue.shutdown();
+ }
+
+ public boolean isEmpty() {
+ return ringBuffer.getBufferSize() == ringBuffer.remainingCapacity();
+ }
+
+ public void setHandlers(IteratorBasedQueueConsumer consumer) {
Review Comment:
I don't think its a good practice to make methods public just for the purpose of testing. Can we please avoid that.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/BoundedInMemoryExecutor.java:
##########
@@ -18,117 +18,97 @@
package org.apache.hudi.common.util.queue;
-import org.apache.hudi.common.util.CustomizedThreadFactory;
import org.apache.hudi.common.util.DefaultSizeEstimator;
import org.apache.hudi.common.util.Functions;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.SizeEstimator;
import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
+import java.io.IOException;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutorCompletionService;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
import java.util.function.Function;
-import java.util.stream.Collectors;
/**
- * Executor which orchestrates concurrent producers and consumers communicating through a bounded in-memory queue. This
+ * Executor which orchestrates concurrent producers and consumers communicating through 'BoundedInMemoryQueue'. This
* class takes as input the size limit, queue producer(s), consumer and transformer and exposes API to orchestrate
* concurrent execution of these actors communicating through a central bounded queue
*/
-public class BoundedInMemoryExecutor<I, O, E> {
+public class BoundedInMemoryExecutor<I, O, E> extends HoodieExecutorBase<I, O, E> {
private static final Logger LOG = LogManager.getLogger(BoundedInMemoryExecutor.class);
- private static final long TERMINATE_WAITING_TIME_SECS = 60L;
- // Executor service used for launching write thread.
- private final ExecutorService producerExecutorService;
- // Executor service used for launching read thread.
- private final ExecutorService consumerExecutorService;
- // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
- private final BoundedInMemoryQueue<I, O> queue;
- // Producers
- private final List<BoundedInMemoryQueueProducer<I>> producers;
- // Consumer
- private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
- // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
- private final Runnable preExecuteRunnable;
+ private final HoodieMessageQueue<I, O> queue;
public BoundedInMemoryExecutor(final long bufferLimitInBytes, final Iterator<I> inputItr,
- BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ IteratorBasedQueueConsumer<O, E> consumer, Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, new IteratorBasedQueueProducer<>(inputItr), Option.of(consumer), transformFunction, preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
this(bufferLimitInBytes, producer, consumer, transformFunction, Functions.noop());
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, BoundedInMemoryQueueProducer<I> producer,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, HoodieProducer<I> producer,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, Runnable preExecuteRunnable) {
this(bufferLimitInBytes, Collections.singletonList(producer), consumer, transformFunction, new DefaultSizeEstimator<>(), preExecuteRunnable);
}
- public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<BoundedInMemoryQueueProducer<I>> producers,
- Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ public BoundedInMemoryExecutor(final long bufferLimitInBytes, List<HoodieProducer<I>> producers,
+ Option<IteratorBasedQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
final SizeEstimator<O> sizeEstimator, Runnable preExecuteRunnable) {
- this.producers = producers;
- this.consumer = consumer;
- this.preExecuteRunnable = preExecuteRunnable;
- // Ensure fixed thread for each producer thread
- this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
- // Ensure single thread for consumer
- this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
- this.queue = new BoundedInMemoryQueue<>(bufferLimitInBytes, transformFunction, sizeEstimator);
+ super(producers, consumer, preExecuteRunnable);
+ this.queue = new BoundedInMemoryQueueIterable<>(bufferLimitInBytes, transformFunction, sizeEstimator);
}
/**
- * Start all Producers.
+ * Start all producers at once.
*/
- public ExecutorCompletionService<Boolean> startProducers() {
+ @Override
+ public CompletableFuture<Void> startProducers() {
// Latch to control when and which producer thread will close the queue
final CountDownLatch latch = new CountDownLatch(producers.size());
- final ExecutorCompletionService<Boolean> completionService =
- new ExecutorCompletionService<Boolean>(producerExecutorService);
- producers.stream().map(producer -> {
- return completionService.submit(() -> {
+
+ return CompletableFuture.allOf(producers.stream().map(producer -> {
+ return CompletableFuture.supplyAsync(() -> {
try {
- preExecuteRunnable.run();
Review Comment:
previously preExecuteRunnable was executed both in producer as well as consumers. W/ this patch, I see we are executing it only for consumers? is that intentional ?
also, I only see it getting executed in BoundedInMemory and not in Distruptor based impl. can you check on that please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1009218766
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieExecutorBase.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HoodieExecutorBase holds common elements producerExecutorService, consumerExecutorService, producers and a single consumer.
+ * Also HoodieExecutorBase control the lifecycle of producerExecutorService and consumerExecutorService.
+ */
+public abstract class HoodieExecutorBase<I, O, E> implements HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieExecutorBase.class);
+
+ private static final long TERMINATE_WAITING_TIME_SECS = 60L;
+ // Executor service used for launching write thread.
+ public final ExecutorService producerExecutorService;
Review Comment:
Sure thing. Just go through all the PR and changed wherever applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283923331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1270259978
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 67d8cd3c14c8539a4033c9e4a99b16eef9a03c25 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008)
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 3e44fb2397bdbfa9c63fab47e62f312451b91900 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271113037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 3e44fb2397bdbfa9c63fab47e62f312451b91900 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029)
* 672fdc6665cb250678c0768f55c8086a3cbfc388 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989699671
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import org.jetbrains.annotations.NotNull;
Review Comment:
No need for that actually, just removed.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieConsumer.java:
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class HoodieConsumer<I, O> {
+
+ /**
+ * Consumer One record.
+ */
+ protected abstract void consumeOneRecord(I record);
+
+ /**
+ * Notifies implementation that we have exhausted consuming records from queue.
+ */
+ protected abstract void finish();
+
+ /**
+ * Return result of consuming records so far.
+ */
+ protected abstract O getResult();
Review Comment:
No need for that actually, just removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989700875
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorBasedProducer.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public abstract class DisruptorBasedProducer<I> implements HoodieProducer<I> {
Review Comment:
Nice catch! totally agree!
abstract HoodieConsumer and HoodieProducer. They modify HoodieMessageQueue and don't care what the specific implementation of HoodieMessageQueue is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989704508
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
+ this.queue = new DisruptorMessageQueue<>(bufferSize, transformFunction, waitStrategy, producers.size(), preExecuteRunnable);
+ }
+
+ /**
+ * Start all Producers.
+ */
+ public ExecutorCompletionService<Boolean> startProducers() {
+ final ExecutorCompletionService<Boolean> completionService =
+ new ExecutorCompletionService<Boolean>(producerExecutorService);
+ producers.stream().map(producer -> {
+ return completionService.submit(() -> {
+ try {
+ preExecuteRunnable.run();
+
+ DisruptorPublisher publisher = new DisruptorPublisher<>(producer, queue);
+ publisher.startProduce();
+
+ } catch (Throwable e) {
+ LOG.error("error producing records", e);
+ throw e;
+ }
+ return true;
+ });
+ }).collect(Collectors.toList());
+ return completionService;
+ }
+
+ @Override
+ public E execute() {
Review Comment:
Sure. Just unify the logic here, in execute both disruptor executor and original bounded Executor will call `startConsumer()`, `startProducers()` and `finishConsuming()` only. Although each has its own implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989705143
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorExecutor.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.dsl.Disruptor;
+import org.apache.hudi.common.util.CustomizedThreadFactory;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorCompletionService;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+public class DisruptorExecutor<I, O, E> extends HoodieExecutor<I, O, E> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorExecutor.class);
+
+ // Executor service used for launching write thread.
+ private final ExecutorService producerExecutorService;
+ // Executor service used for launching read thread.
+ private final ExecutorService consumerExecutorService;
+ // Used for buffering records which is controlled by HoodieWriteConfig#WRITE_BUFFER_LIMIT_BYTES.
+ private final DisruptorMessageQueue<I, O> queue;
+ // Producers
+ private final List<DisruptorBasedProducer<I>> producers;
+ // Consumer
+ private final Option<BoundedInMemoryQueueConsumer<O, E>> consumer;
+ // pre-execute function to implement environment specific behavior before executors (producers/consumer) run
+ private final Runnable preExecuteRunnable;
+
+ public DisruptorExecutor(final int bufferSize, final Iterator<I> inputItr,
+ BoundedInMemoryQueueConsumer<O, E> consumer, Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, new IteratorBasedDisruptorProducer<>(inputItr), Option.of(consumer), transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction) {
+ this(bufferSize, producer, consumer, transformFunction, WaitStrategyFactory.DEFAULT_STRATEGY, Functions.noop());
+ }
+
+ public DisruptorExecutor(final int bufferSize, DisruptorBasedProducer<I> producer,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction, String waitStrategy, Runnable preExecuteRunnable) {
+ this(bufferSize, Collections.singletonList(producer), consumer, transformFunction, waitStrategy, preExecuteRunnable);
+ }
+
+ public DisruptorExecutor(final int bufferSize, List<DisruptorBasedProducer<I>> producers,
+ Option<BoundedInMemoryQueueConsumer<O, E>> consumer, final Function<I, O> transformFunction,
+ final String waitStrategy, Runnable preExecuteRunnable) {
+ this.producers = producers;
+ this.consumer = consumer;
+ this.preExecuteRunnable = preExecuteRunnable;
+ // Ensure fixed thread for each producer thread
+ this.producerExecutorService = Executors.newFixedThreadPool(producers.size(), new CustomizedThreadFactory("producer"));
+ // Ensure single thread for consumer
+ this.consumerExecutorService = Executors.newSingleThreadExecutor(new CustomizedThreadFactory("consumer"));
Review Comment:
Since we now have HoodieExecutorBase it holds common elements producerExecutorService, consumerExecutorService, producers and consumer and share the same logic for creating consumerExecutorService here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271305720
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* c0c22741ed123bd13cff21dec9fb7efdc70b99f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042)
* 4587303118918c5e56ecb10732d9fcba43a90ee7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r989934770
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -230,6 +240,16 @@ public class HoodieWriteConfig extends HoodieConfig {
.defaultValue(String.valueOf(4 * 1024 * 1024))
.withDocumentation("Size of in-memory buffer used for parallelizing network reads and lake storage writes.");
+ public static final ConfigProperty<Integer> WRITE_BUFFER_SIZE = ConfigProperty
+ .key("hoodie.write.buffer.size")
+ .defaultValue(1024)
+ .withDocumentation("The size of the Disruptor Executor ring buffer, must be power of 2");
+
+ public static final ConfigProperty<String> WRITE_WAIT_STRATEGY = ConfigProperty
+ .key("hoodie.write.wait.strategy")
+ .defaultValue("BlockingWaitStrategy")
Review Comment:
Sure, just remove the "Strategy" suffix :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1271613867
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 4ba91d4ce8345b4917e1f402694a55d07bf2951c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1283623679
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1282483705
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1288416366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12250",
"triggerID" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"triggerType" : "PUSH"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12254",
"triggerID" : "1280769517",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1280628444",
"triggerType" : "MANUAL"
}, {
"hash" : "c748320fb229a7a013f4710ac1a2d68e60175d2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12259",
"triggerID" : "1281167180",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12260",
"triggerID" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"triggerType" : "PUSH"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281684865",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12268",
"triggerID" : "1281731671",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12278",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12270",
"triggerID" : "1281789564",
"triggerType" : "MANUAL"
}, {
"hash" : "3f3a41a03b2fe436a4014e6e6627aa90b7cfd80b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12303",
"triggerID" : "1282483705",
"triggerType" : "MANUAL"
}, {
"hash" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12319",
"triggerID" : "9fe6bdae8829ea7a58d680cee1ea9c870d3549c8",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324",
"triggerID" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"triggerType" : "PUSH"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331",
"triggerID" : "1283633132",
"triggerType" : "MANUAL"
}, {
"hash" : "a81ffdf9c24a3f2f984161ca193af3b387b1e9a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348",
"triggerID" : "1284213374",
"triggerType" : "MANUAL"
}, {
"hash" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506",
"triggerID" : "e5c17f060235551dd9130e7bc7bbc33b294ebb18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* a81ffdf9c24a3f2f984161ca193af3b387b1e9a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12324) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12331) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12348)
* e5c17f060235551dd9130e7bc7bbc33b294ebb18 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12506)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r1003974164
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent> ringBuffer;
+
+ public DisruptorMessageQueue(Option<Integer> bufferSize, Function<I, O> transformFunction, Option<String> waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(new HoodieDisruptorEventFactory(), bufferSize.get(), threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent> translator = new EventTranslator<HoodieDisruptorEvent>() {
+ @Override
+ public void translateTo(HoodieDisruptorEvent event, long sequence) {
+ event.set(applied);
+ }
+ };
+
+ queue.getRingBuffer().publishEvent(translator);
+ }
+
+ @Override
+ public Option<O> readNextRecord() {
+ // Let DisruptorMessageHandler to handle consuming logic.
+ return null;
+ }
+
+ @Override
+ public void markAsFailed(Throwable e) {
+ // do nothing.
+ }
+
+ @Override
+ public void close() {
+ // Waits until all events currently in the disruptor have been processed by all event processors
+ queue.shutdown();
+ }
+
+ public boolean isEmpty() {
+ return ringBuffer.getBufferSize() == ringBuffer.remainingCapacity();
+ }
+
+ public void setHandlers(IteratorBasedQueueConsumer consumer) {
Review Comment:
It need to be used in TestDisruptorMessageQueue#testCompositeProducerRecordReading :<
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java:
##########
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+import com.lmax.disruptor.EventHandler;
+import com.lmax.disruptor.EventTranslator;
+import com.lmax.disruptor.RingBuffer;
+import com.lmax.disruptor.WaitStrategy;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import org.apache.hudi.common.util.Option;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.function.Function;
+
+public class DisruptorMessageQueue<I, O> implements HoodieMessageQueue<I, O> {
+
+ private static final Logger LOG = LogManager.getLogger(DisruptorMessageQueue.class);
+
+ private final Disruptor<HoodieDisruptorEvent> queue;
+ private final Function<I, O> transformFunction;
+ private final RingBuffer<HoodieDisruptorEvent> ringBuffer;
+
+ public DisruptorMessageQueue(Option<Integer> bufferSize, Function<I, O> transformFunction, Option<String> waitStrategyName, int totalProducers, Runnable preExecuteRunnable) {
+ WaitStrategy waitStrategy = WaitStrategyFactory.build(waitStrategyName);
+ HoodieDaemonThreadFactory threadFactory = new HoodieDaemonThreadFactory(preExecuteRunnable);
+
+ this.queue = new Disruptor<>(new HoodieDisruptorEventFactory(), bufferSize.get(), threadFactory, totalProducers > 1 ? ProducerType.MULTI : ProducerType.SINGLE, waitStrategy);
+ this.ringBuffer = queue.getRingBuffer();
+ this.transformFunction = transformFunction;
+ }
+
+ @Override
+ public long size() {
+ return queue.getBufferSize();
+ }
+
+ @Override
+ public void insertRecord(I value) throws Exception {
+ O applied = transformFunction.apply(value);
+
+ EventTranslator<HoodieDisruptorEvent> translator = new EventTranslator<HoodieDisruptorEvent>() {
Review Comment:
Sure thing Changed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1289531298
@zhangyue19921010 please rebase on the latest master. Test failures you observe have already been addressed on master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1278838321
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 447cb4510301af1c3ff1aebb3bd0a668872fc3f6 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069)
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* e230fbb946d8f33a4691129893e2559a3d8a1768 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996575064
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorPublisher.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+public class DisruptorPublisher<I, O> {
Review Comment:
Nice catch. No need for this actually. Removed.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDaemonThreadFactory.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Access to a ThreadFactory instance.
+ * 1. All threads are created with setDaemon(true).
+ * 2. All threads execute preExecuteRunnable func once.
+ */
+public class HoodieDaemonThreadFactory implements ThreadFactory {
+
+ private final Runnable preExecuteRunnable;
+ private final AtomicInteger threadsNum = new AtomicInteger();
+ private final String namePattern;
+ private final String baseName = "Hoodie-disruptor-daemon-thread";
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996552089
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2274,6 +2306,11 @@ public Builder withKeyGenerator(String keyGeneratorClass) {
return this;
}
+ public Builder withExecutorName(String executorClass) {
Review Comment:
Sure, changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on code in PR #5416:
URL: https://github.com/apache/hudi/pull/5416#discussion_r996584105
##########
hudi-common/src/main/java/org/apache/hudi/common/util/queue/HoodieDisruptorEventFactory.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util.queue;
+
+import com.lmax.disruptor.EventFactory;
+
+/**
+ * HoodieDisruptorEventFactory is used to create/preallocate HoodieDisruptorEvent.
+ *
+ */
+public class HoodieDisruptorEventFactory<O> implements EventFactory<HoodieDisruptorEvent<O>> {
Review Comment:
All move for HoodieDisruptorEvent and HoodieDisruptorEventFactory into `DisruptorMessageQueue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280534000
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245)
* d615f1f02d90ce8a26c3a9c7c0fd479e89de3155 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280526199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b838e1f406902c9bdfb5e84d53ef5a5effd0765b",
"triggerType" : "PUSH"
}, {
"hash" : "3778cf46256d3efb36646b29f06cf149e839045c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8269",
"triggerID" : "3778cf46256d3efb36646b29f06cf149e839045c",
"triggerType" : "PUSH"
}, {
"hash" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9102",
"triggerID" : "8041a028f3f28b8d4745753b3ffffc452002b34e",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"triggerType" : "PUSH"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1162845670",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1163021664",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1166739686",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1173619059",
"triggerType" : "MANUAL"
}, {
"hash" : "1b868643f2f8794b4dd47f8e36124749df6effa3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=9446",
"triggerID" : "1177380885",
"triggerType" : "MANUAL"
}, {
"hash" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6114ee2aa59f087e5ef0b1b53979eec143b33f5e",
"triggerType" : "PUSH"
}, {
"hash" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12008",
"triggerID" : "67d8cd3c14c8539a4033c9e4a99b16eef9a03c25",
"triggerType" : "PUSH"
}, {
"hash" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92760dbf5a047fe1f9941fa4b36c944eb3bec5c7",
"triggerType" : "PUSH"
}, {
"hash" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12029",
"triggerID" : "3e44fb2397bdbfa9c63fab47e62f312451b91900",
"triggerType" : "PUSH"
}, {
"hash" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12040",
"triggerID" : "672fdc6665cb250678c0768f55c8086a3cbfc388",
"triggerType" : "PUSH"
}, {
"hash" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12042",
"triggerID" : "c0c22741ed123bd13cff21dec9fb7efdc70b99f8",
"triggerType" : "PUSH"
}, {
"hash" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12044",
"triggerID" : "4587303118918c5e56ecb10732d9fcba43a90ee7",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12047",
"triggerID" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"triggerType" : "PUSH"
}, {
"hash" : "4ba91d4ce8345b4917e1f402694a55d07bf2951c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12052",
"triggerID" : "1271688530",
"triggerType" : "MANUAL"
}, {
"hash" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12069",
"triggerID" : "447cb4510301af1c3ff1aebb3bd0a668872fc3f6",
"triggerType" : "PUSH"
}, {
"hash" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0fc24d3af4181d8fb68d803b97be78e5cd448787",
"triggerType" : "PUSH"
}, {
"hash" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12211",
"triggerID" : "e230fbb946d8f33a4691129893e2559a3d8a1768",
"triggerType" : "PUSH"
}, {
"hash" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1",
"triggerType" : "PUSH"
}, {
"hash" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241",
"triggerID" : "794e30be06f056e60824952e06c7eb7de4eb50c8",
"triggerType" : "PUSH"
}, {
"hash" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245",
"triggerID" : "6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39",
"triggerType" : "PUSH"
}, {
"hash" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d615f1f02d90ce8a26c3a9c7c0fd479e89de3155",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b838e1f406902c9bdfb5e84d53ef5a5effd0765b UNKNOWN
* 6114ee2aa59f087e5ef0b1b53979eec143b33f5e UNKNOWN
* 92760dbf5a047fe1f9941fa4b36c944eb3bec5c7 UNKNOWN
* 0fc24d3af4181d8fb68d803b97be78e5cd448787 UNKNOWN
* 298f66d2842b1fa3ce9c487fd3d0f94eda4bd2b1 UNKNOWN
* 794e30be06f056e60824952e06c7eb7de4eb50c8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12241)
* 6e56bd1d5d3a07cde1b23c66985b4cb4cc85ee39 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12245)
* d615f1f02d90ce8a26c3a9c7c0fd479e89de3155 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #5416: [HUDI-3963] Use Lock-Free Message Queue Disruptor Improving Hoodie Writing Efficiency
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on PR #5416:
URL: https://github.com/apache/hudi/pull/5416#issuecomment-1280628444
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org