You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "iamssx (via GitHub)" <gi...@apache.org> on 2023/03/02 07:46:49 UTC
[GitHub] [rocketmq] iamssx opened a new issue, #6226: a large number of FlowMonitor thread in rocketmq
iamssx opened a new issue, #6226:
URL: https://github.com/apache/rocketmq/issues/6226
use the default config to start up a rocketmq cluster,when it run for a month or so, there will be a jvm crash because of oom. we try to modify the linux memory arguments , and that is not worked. and we get the jvm dump and thread stack. now we found that a lot of FlowMonitor thread wating, and it keep increase.we suspect the amount of flowmonitor thread is the reason of jvm crash.
please read the thread analysis and jvm crash, and tell us the true reason why it crash, and the way to fix it, thanks!
environment:
rocketmq 5.0.0
linux redhat7.9, 48Core/755GB MEM
/usr/local/jdk/bin/java -server -Xms8g -Xmx8g -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0 -verbose:gc -Xloggc:/dev/shm/rmq_srv_gc_%p_%t.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:MaxDirectMemorySize=15g -XX:-UseLargePages -XX:-UseBiasedLocking -Drocketmq.client.logUseSlf4j=true -cp .:/data/rocketmq-all-5.0.0-bin-release//bin/../conf:/data/rocketmq-all-5.0.0-bin-release//bin/../lib/*: -Djdk.tls.rejectClientInitiatedRenegotiation=true org.apacherocketmq.broker.BrokerStartup -c /data/rocketmq/conf/broker-c.properties
the jvm crash info:
thread dump:
https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjMvMDMvMi90aHJlYWQudHh0LS03LTE0LTU2&
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] guyinyou commented on issue #6226: a large number of FlowMonitor thread in rocketmq
Posted by "guyinyou (via GitHub)" <gi...@apache.org>.
guyinyou commented on issue #6226:
URL: https://github.com/apache/rocketmq/issues/6226#issuecomment-1451571710
this is a great find, It is caused by no shutdown when connection closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on issue #6226: a large number of FlowMonitor thread in rocketmq
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on issue #6226:
URL: https://github.com/apache/rocketmq/issues/6226#issuecomment-1451548337
Hi @iamssx, I wonder know whether the oom is master node or slave node? Is it controller mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamssx commented on issue #6226: a large number of FlowMonitor thread in rocketmq
Posted by "iamssx (via GitHub)" <gi...@apache.org>.
iamssx commented on issue #6226:
URL: https://github.com/apache/rocketmq/issues/6226#issuecomment-1452793975
> Hi @iamssx I have reproduce the issue and will fix it ASAP.
thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on issue #6226: a large number of FlowMonitor thread in rocketmq
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin commented on issue #6226:
URL: https://github.com/apache/rocketmq/issues/6226#issuecomment-1451685446
Hi @iamssx I have reproduce the issue and will fix it ASAP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin closed issue #6226: a large number of FlowMonitor thread in rocketmq
Posted by "RongtongJin (via GitHub)" <gi...@apache.org>.
RongtongJin closed issue #6226: a large number of FlowMonitor thread in rocketmq
URL: https://github.com/apache/rocketmq/issues/6226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] iamssx commented on issue #6226: a large number of FlowMonitor thread in rocketmq
Posted by "iamssx (via GitHub)" <gi...@apache.org>.
iamssx commented on issue #6226:
URL: https://github.com/apache/rocketmq/issues/6226#issuecomment-1451437942
there is enougth free memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org