You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by "lazyCoder10 (via GitHub)" <gi...@apache.org> on 2023/07/12 00:44:10 UTC
[GitHub] [rocketmq] lazyCoder10 created a discussion: RocketMQ is unable to handle request size of 10Kb
GitHub user lazyCoder10 created a discussion: RocketMQ is unable to handle request size of 10Kb
For an academic project, we are benchmarking RocketMQ with the OpenMessagingBenchmark Framework (https://openmessaging.cloud/docs/benchmarks/). When running the simulation for 10Kb message size with 1-topic-1partition configuration, the benchmarking tool is throwing the below error:
16:07:36.962 [local-worker-1-1] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.rocketmq.client.exception.MQClientException: executor rejected
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:813) ~[?:?]
at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803) ~[?:?]
at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195) ~[?:?]
at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:47) ~[classes/:?]
at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$6(LocalWorker.java:206) ~[classes/:?]
at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$7(LocalWorker.java:204) ~[classes/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.rocketmq.client.exception.MQClientException: executor rejected
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:519) ~[rocketmq-client-4.9.3.jar:4.9.3]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:482) ~[rocketmq-client-4.9.3.jar:4.9.3]
at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:376) ~[rocketmq-client-4.9.3.jar:4.9.3]
at io.openmessaging.benchmark.driver.rocketmq.RocketMQBenchmarkProducer.sendAsync(RocketMQBenchmarkProducer.java:43) ~[classes/:?]
at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:46) ~[classes/:?]
... 9 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2f39d323[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@22dc1731[Wrapped task = org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl$3@cf5f484]] rejected from java.util.concurrent.ThreadPoolExecutor@312a63fb[Running, pool size = 16, active threads = 16, queued tasks = 50000, completed tasks = 118613]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2070) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) ~[?:?]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123) ~[?:?]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:501) ~[rocketmq-client-4.9.3.jar:4.9.3]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:482) ~[rocketmq-client-4.9.3.jar:4.9.3]
at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:376) ~[rocketmq-client-4.9.3.jar:4.9.3]
at io.openmessaging.benchmark.driver.rocketmq.RocketMQBenchmarkProducer.sendAsync(RocketMQBenchmarkProducer.java:43) ~[classes/:?]
at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:46) ~[classes/:?]
... 9 more
But when we reduced the message size to 1kb with the same topic and partition configuration, it works as expected. To resolve the issue I have followed the following steps:
- Increased the JVM memory to 24G for both broker and nameserver. Primarily my assumption was the error is due to memory starvation of JVM. But after increasing the memory the problem still remains. All the brokers have 32G of memory and 8 Core CPU.
- Changed the adminBrokerThreadPoolNums to 100000 in broker properties. We are using 3-master-no-slave configuration. But the problem still remains.
GitHub link: https://github.com/apache/rocketmq/discussions/7015
----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org
Re: [D] RocketMQ is unable to handle request size of 10Kb [rocketmq]
Posted by "mymx2 (via GitHub)" <gi...@apache.org>.
GitHub user mymx2 added a comment to the discussion: RocketMQ is unable to handle request size of 10Kb
focus:
- [Comparative Evaluation of Java Virtual Machine-Based Message Queue Services: A Study on Kafka, Artemis, Pulsar, and RocketMQ](https://www.mdpi.com/2079-9292/12/23/4792)
- #9452
GitHub link: https://github.com/apache/rocketmq/discussions/7015#discussioncomment-13427649
----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org
[GitHub] [rocketmq] lazyCoder10 added a comment to the discussion: RocketMQ is unable to handle request size of 10Kb
Posted by "lazyCoder10 (via GitHub)" <gi...@apache.org>.
GitHub user lazyCoder10 added a comment to the discussion: RocketMQ is unable to handle request size of 10Kb
Actually we are comparing several JVM based MQs to learn how the MQs behave in different scenario. If we rate limit for rocketMq, won't that be an unfair advantage?
Without limiting the tps to a lower level, what would be your suggestion? We need to run load test for 10Kb, 100Kb, 1000Kb message size.
GitHub link: https://github.com/apache/rocketmq/discussions/7015#discussioncomment-6432088
----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org
[GitHub] [rocketmq] GenerousMan added a comment to the discussion: RocketMQ is unable to handle request size of 10Kb
Posted by "GenerousMan (via GitHub)" <gi...@apache.org>.
GitHub user GenerousMan added a comment to the discussion: RocketMQ is unable to handle request size of 10Kb
Maybe 10KB will put a lot of pressure on the broker under the tps set in openmessaging, resulting in the rejection of some message sending requests.
Appropriately reducing the sending rate may solve this problem, or we need to turn on backpressure in such occasions.
GitHub link: https://github.com/apache/rocketmq/discussions/7015#discussioncomment-6425354
----
This is an automatically sent email for dev@rocketmq.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@rocketmq.apache.org