You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/11/16 09:31:41 UTC
[GitHub] [rocketmq] xiaobing395845388 opened a new issue, #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
xiaobing395845388 opened a new issue, #5417:
URL: https://github.com/apache/rocketmq/issues/5417
The issue tracker is used for bug reporting purposes **ONLY** whereas feature request needs to follow the [RIP process](https://github.com/apache/rocketmq/wiki/RocketMQ-Improvement-Proposal). To avoid unnecessary duplication, please check whether there is a previous issue before filing a new one.
It is recommended to start a discussion thread in the [mailing lists](http://rocketmq.apache.org/about/contact/) in cases of discussing your deployment plan, API clarification, and other non-bug-reporting issues.
We welcome any friendly suggestions, bug fixes, collaboration, and other improvements.
Please ensure that your bug report is clear and self-contained. Otherwise, it would take additional rounds of communication, thus more time, to understand the problem itself.
Generally, fixing an issue goes through the following steps:
1. Understand the issue reported;
1. Reproduce the unexpected behavior locally;
1. Perform root cause analysis to identify the underlying problem;
1. Create test cases to cover the identified problem;
1. Work out a solution to rectify the behavior and make the newly created test cases pass;
1. Make a pull request and go through peer review;
As a result, it would be very helpful yet challenging if you could provide an isolated project reproducing your reported issue. Anyway, please ensure your issue report is informative enough for the community to pick up. At a minimum, include the following hints:
**BUG REPORT**
1. Please describe the issue you observed:
in this case,consumeThreadMin work as coresize, and consumeThreadMax does work and my cause oom error in high flow
2. Please tell us about your environment:
3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):
**FEATURE REQUEST**
1. Please describe the feature you are requesting.
2. Provide any additional detail on your proposed use case for this feature.
2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
4. If there are some sub-tasks involved, use -[] for each sub-task and create a corresponding issue to map to the sub-task:
- [sub-task1-issue-number](example_sub_issue1_link_here): sub-task1 description here,
- [sub-task2-issue-number](example_sub_issue2_link_here): sub-task2 description here,
- ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] xiaobing395845388 commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
xiaobing395845388 commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1294834731
in this case,consumeThreadMin work as coresize, and consumeThreadMax does work and my cause oom error in high flow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor [rocketmq]
Posted by "sunheyi6 (via GitHub)" <gi...@apache.org>.
sunheyi6 commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1815654929
> The key point what i want to express is not 'oom' but only 'consumeThreadMin 'works as corepoolsize, param 'consumeThreadMax 'does not work我想表达的关键点不是'oom'而是只有'consumeThreadMin'作为corepoolsize起作用，param'consumeThreadMax'不起作用
Will it be because the consumer's workload is insufficient to require additional threads, so consumeThreadMax does not take effect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] xiaobing395845388 closed issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
xiaobing395845388 closed issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
URL: https://github.com/apache/rocketmq/issues/5417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] fuyou001 commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
fuyou001 commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1313635869
@xiaobing395845388 ,what's your application JVM arguments about jvm memory setting?
if you want to avoid oom ,you can set DefaultMQPushConsumer arguments
```
org.apache.rocketmq.client.consumer.DefaultMQPushConsumer#pullThresholdForTopic
org.apache.rocketmq.client.consumer.DefaultMQPushConsumer#pullThresholdForQueue
org.apache.rocketmq.client.consumer.DefaultMQPushConsumer#pullThresholdSizeForTopic
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] pipi18 commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
pipi18 commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1296217745
Consumer use `LinkedBlockingQueue ` may trigger flow control to aviod OOM. When messages are greater or equal than 1000, or the the memory is greater 100M.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] nowinkeyy commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
nowinkeyy commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1294990658
Hi @xiaobing395845388 ,
IMO, you mean unbounded ```LinkedBlockingQueue```, and ```consumeThreadMax``` is too small resulting in OOM.
My suggestion is to increase the value of ```consumeThreadMax``` to speed up message processing and decrease the value of ```pullThresholdForQueue``` to reduce the number of messages cached by the client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lizhanhui commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
lizhanhui commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1313640796
@fuyou001 How about making these configurable items heuristic, say, adjust them according to JVM heap size on start-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lizhanhui commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
lizhanhui commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1313643161
We essentially need a smarter back-pressure mechanism to protect against memory thrashing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] shy-share commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
shy-share commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1294997391
> Hi @xiaobing395845388 , IMO, you mean unbounded `LinkedBlockingQueue`, and `consumeThreadMax` is too small resulting in OOM. My suggestion is to increase the value of `consumeThreadMax` to speed up message processing and decrease the value of `pullThresholdForQueue` to reduce the number of messages cached by the client.
@xiaobing395845388 Indeed, LinkedBlockingQueue is suitable for situations where producing messages and consuming messages are about the same, and when you produce messages significantly faster than you consume them, it naturally leads to oom
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] xiaobing395845388 commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
xiaobing395845388 commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1296436873
>
> > Hi @xiaobing395845388 , IMO, you mean unbounded `LinkedBlockingQueue`, and `consumeThreadMax` is too small resulting in OOM. My suggestion is to increase the value of `consumeThreadMax` to speed up message processing and decrease the value of `pullThresholdForQueue` to reduce the number of messages cached by the client.
>
> @xiaobing395845388 Indeed, LinkedBlockingQueue is suitable for situations where producing messages and consuming messages are about the same, and when you produce messages significantly faster than you consume them, it naturally leads to oom
In this case, consumeThreadMax does no work if use LinkedBlockingQueue , consumeThreadMIn is the corePoolSize in the api of "ThreadPoolExecutor"
you say"producing messages and consuming messages are about the same", we cannot controll this in product env
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] xiaobing395845388 commented on issue #5417: consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor
Posted by GitBox <gi...@apache.org>.
xiaobing395845388 commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1316229153
The key point that i want to express is not 'oom' but only 'consumeThreadMin 'works as corepoolsize, param 'consumeThreadMax 'does not work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] consumeExecutor use LinkedBlockingQueue which is not bounded as param of ThreadPoolExecutor [rocketmq]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5417:
URL: https://github.com/apache/rocketmq/issues/5417#issuecomment-1815524182
This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org