You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/01/21 14:50:42 UTC
[GitHub] [rocketmq] lzhang131 opened a new pull request #3784: #3781 压测生产者内存溢出
lzhang131 opened a new pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784
**Make sure set the target branch to `develop`**
## What is the purpose of the change
#3781 When the producer sends a message successfully, the thread pool is newFixedThreadPool. The cache queue is unbounded and large concurrency causes memory overflow.
## Brief changelog
The custom thread pool sets the bounded queue length to 50000
close #3781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] XiaoyiPeng commented on a change in pull request #3784: #3781 压测生产者内存溢出
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng commented on a change in pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784#discussion_r789733049
##########
File path: remoting/src/main/java/org/apache/rocketmq/remoting/netty/NettyRemotingClient.java
##########
@@ -111,14 +108,21 @@ public NettyRemotingClient(final NettyClientConfig nettyClientConfig,
publicThreadNums = 4;
}
- this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
- private AtomicInteger threadIndex = new AtomicInteger(0);
+ this.publicThreadPoolQueue = new LinkedBlockingQueue<Runnable>(50000);
Review comment:
I don't think that's the problem here, `publicExecutor` handles requests from the server (eg: `CHECK_TRANSACTION_STATE` or `RESET_CONSUMER_CLIENT_OFFSET` etc) instead of client send requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lzhang131 commented on pull request #3784: #3781 压测生产者内存溢出
Posted by GitBox <gi...@apache.org>.
lzhang131 commented on pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784#issuecomment-1019082837
> The `callbackExecutor` has been redefined in issue #3781 , and its queue is bounded with 50000 as you defined in your commit.
>
> So the `publicExecutor` will not be used as `callbackExecutor` when called `getCallbackExecutor()`.
>
> ```java
> @Override
> public ExecutorService getCallbackExecutor() {
> return callbackExecutor != null ? callbackExecutor : publicExecutor;
> }
> ```
>
> So I think your PR can not solve the issue #3781
You can look at this code：
private void executeInvokeCallback(final ResponseFuture responseFuture) {
boolean runInThisThread = false;
ExecutorService executor = this.getCallbackExecutor();
if (executor != null) {
try {
executor.submit(new Runnable() {
@Override
public void run() {
try {
responseFuture.executeInvokeCallback();
} catch (Throwable e) {
log.warn("execute callback in executor exception, and callback throw", e);
} finally {
responseFuture.release();
}
}
});
} catch (Exception e) {
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] XiaoyiPeng commented on pull request #3784: #3781 压测生产者内存溢出
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng commented on pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784#issuecomment-1019026376
The `callbackExecutor` has been redefined in issue #3781 , and its queue is bounded with 50000 as you defined in your commit.
So the `publicExecutor` will not be used as `callbackExecutor` when called `getCallbackExecutor()`.
```java
@Override
public ExecutorService getCallbackExecutor() {
return callbackExecutor != null ? callbackExecutor : publicExecutor;
}
```
So I think your PR can not solve the issue #3781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] XiaoyiPeng commented on a change in pull request #3784: #3781 压测生产者内存溢出
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng commented on a change in pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784#discussion_r789733049
##########
File path: remoting/src/main/java/org/apache/rocketmq/remoting/netty/NettyRemotingClient.java
##########
@@ -111,14 +108,21 @@ public NettyRemotingClient(final NettyClientConfig nettyClientConfig,
publicThreadNums = 4;
}
- this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
- private AtomicInteger threadIndex = new AtomicInteger(0);
+ this.publicThreadPoolQueue = new LinkedBlockingQueue<Runnable>(50000);
Review comment:
I don't think that's the problem here, `publicExecutor` handles requests from the server (eg: `CHECK_TRANSACTION_STATE` or `RESET_CONSUMER_CLIENT_OFFSET` etc) instead of client send requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] guyinyou commented on pull request #3784: #3781 压测生产者内存溢出
Posted by GitBox <gi...@apache.org>.
guyinyou commented on pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784#issuecomment-1018697226
If more than 50000, is there a rejection policy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lzhang131 commented on pull request #3784: #3781 压测生产者内存溢出
Posted by GitBox <gi...@apache.org>.
lzhang131 commented on pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784#issuecomment-1019082371
> If more than 50000, is there a rejection policy
Changes Rejection Policy Resubmitted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lzhang131 closed pull request #3784: #3781 压测生产者内存溢出
Posted by GitBox <gi...@apache.org>.
lzhang131 closed pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] lzhang131 commented on pull request #3784: #3781 压测生产者内存溢出
Posted by GitBox <gi...@apache.org>.
lzhang131 commented on pull request #3784:
URL: https://github.com/apache/rocketmq/pull/3784#issuecomment-1019082511
> The `callbackExecutor` has been redefined in issue #3781 , and its queue is bounded with 50000 as you defined in your commit.
>
> So the `publicExecutor` will not be used as `callbackExecutor` when called `getCallbackExecutor()`.
>
> ```java
> @Override
> public ExecutorService getCallbackExecutor() {
> return callbackExecutor != null ? callbackExecutor : publicExecutor;
> }
> ```
>
> So I think your PR can not solve the issue #3781
> ```java
> publicExecutor
> ```
If you don't set the default, you will use this publicExecutor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org