You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/01/21 10:17:21 UTC

[GitHub] [rocketmq] yylstudy opened a new issue #3781: 压测生产者内存溢出

yylstudy opened a new issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781


   1. Please describe the issue you observed:
   压测生产者内存溢出
   2. Please tell us about your environment:
   操作系统：centos
   broker版本：v_4.9.2
   rocketmq-client版本：4.9.2
   
   创建生产者代码
   
   asyncSenderExecutor 80个线程，队列长度为50000，线程池拒绝策略为CallerRunsPolicy
   callbackExecutor，80个线程，队列长度为50000，线程池拒绝策略为：AbortPolicy
   ![image](https://user-images.githubusercontent.com/26402953/150509710-c29b6ac1-e498-4a7c-9875-0b2c7e0fb8dd.png)
   
   异步发送消息
   ![image](https://user-images.githubusercontent.com/26402953/150507857-9f747543-2613-4b59-bf20-ea2f015d3248.png)
   
   当压测到1亿数据量时，生产消息很慢
   查看生产者日志，invokeSync call timeout错误
   
   查看生产者gc时间，发现一直在fullgc
   ![image](https://user-images.githubusercontent.com/26402953/150508149-fbef16ef-9229-4996-8277-0e8bd161a21e.png)
   
   导出dump文件，发现大量的CompletableFuture对象
   ![image](https://user-images.githubusercontent.com/26402953/150508712-593414c3-2deb-482f-bee0-e215e79397ed.png)
   
   请问下这个怎么解决，是不是因为单个客户端请求量太大了导致回调队列太长，但是我配置了
   callbackExecutor 线程池拒绝策略为：AbortPolicy 也没有效果。请问下怎么解决呢 谢谢
   
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] XiaoyiPeng edited a comment on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
XiaoyiPeng edited a comment on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1019024783


   > callbackExecutor，2cpu个线程，队列长度为50000，线程池拒绝策略为：AbortPolicy
   
   The `AbortPolicy` has no effect for `callbackExecutor`, because when `callbackExecutor` catch `RejectedExecutionException` in the external invoked thread, it does not throw  exception to the upper layer.
   
   As shown below: 
   
   ![image](https://user-images.githubusercontent.com/8653312/150621993-ddbef5d8-0856-41d6-801a-b121d065c475.png)
   
   
   
   
   **So no matter which rejection policy is set for** `callbackExecutor`, **the final real policy is the** `CallerRunsPolicy`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] XiaoyiPeng commented on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
XiaoyiPeng commented on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1018575843


   Can you show the log of your application? 
   The more detailed the better, especially the error stack and `caused by`  information


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] yylstudy closed issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
yylstudy closed issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] XiaoyiPeng edited a comment on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
XiaoyiPeng edited a comment on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1019024783


   > callbackExecutor，2cpu个线程，队列长度为50000，线程池拒绝策略为：AbortPolicy
   
   The `AbortPolicy` has no effect for `callbackExecutor`, because when `callbackExecutor` catch `RejectedExecutionException` in the external invoked thread, it does not throw the exception to the upper layer.
   
   As shown below: 
   
   ![image](https://user-images.githubusercontent.com/8653312/150621993-ddbef5d8-0856-41d6-801a-b121d065c475.png)
   
   
   
   
   **So no matter which rejection policy is set for** `callbackExecutor`, **the final real policy is the** `CallerRunsPolicy`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] yylstudy commented on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
yylstudy commented on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1019044166


   If callbackexecutor is not set，value is publicExecutor
   ![image](https://user-images.githubusercontent.com/26402953/150624416-669d9dfc-912d-4243-b2f2-d48a6880eabe.png)
   publicExecutor workQueue is LinkedBlockingQueue of Integer.MAX_VALUE
   ![image](https://user-images.githubusercontent.com/26402953/150624421-5ec0266a-b86c-4514-9b31-f865fe45f423.png)
   
   Is he the cause of frequent fullgc?  thanks
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] yylstudy commented on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
yylstudy commented on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1019049305


   这个是压测的场景，每秒100000的消息量，我想一个客户端压个几小时，但是压到一半生产者就一直fullgc。
   This is the scenario of pressure measurement, with 100000 messages per second. I want to press a client for a few hours, but when it is pressed to half, the producer keeps fullgc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] yylstudy commented on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
yylstudy commented on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1019048582


   The following is the error log
   ![image](https://user-images.githubusercontent.com/26402953/150624806-7f5000a6-64b8-40c7-a6f4-08ad2d8fce45.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] yylstudy commented on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
yylstudy commented on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1019055671


   Sorry, it's my producer code that caused oom. It's a waste of your time. I'll close this issue thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] XiaoyiPeng commented on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
XiaoyiPeng commented on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1019024783


   > callbackExecutor，2cpu个线程，队列长度为50000，线程池拒绝策略为：AbortPolicy
   
   The `AbortPolicy` has no effect for `callbackExecutor`, because when `callbackExecutor` catch `RejectedExecutionException`, the external invoked thread, without throwing exceptions to the upper layer.
   
   As shown below: 
   
   ![image](https://user-images.githubusercontent.com/8653312/150621993-ddbef5d8-0856-41d6-801a-b121d065c475.png)
   
   
   
   
   **So no matter which rejection policy is set for** `callbackExecutor`, **the final real policy is the** `CallerRunsPolicy`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] yylstudy removed a comment on issue #3781: 压测生产者内存溢出

Posted by GitBox <gi...@apache.org>.
yylstudy removed a comment on issue #3781:
URL: https://github.com/apache/rocketmq/issues/3781#issuecomment-1019048582






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org