You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2020/07/02 12:23:30 UTC
[GitHub] [rocketmq] kayyellow opened a new issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
kayyellow opened a new issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132
1. Hardware configuration
8C - 32G - aliyun ecs 高效云盘
2. The configuration for broker：
`brokerClusterName=DefaultCluster
brokerName=broker-b
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
#发送消息的最大线程数
sendMessageThreadPoolNums: 24
#发送消息是否使用可重入锁
useReentrantLockWhenPutMessage: true
waitTimeMillsInSendQueue=1000
mapedFileSizeCommitLog=1073741824
storePathRootDir=/usr/local/rocketmq/store
storePathCommitLog=/usr/local/rocketmq/store/commitlog
storePathConsumerQueue=/usr/local/rocketmq/store/consumequeue
storePathIndex=/usr/local/rocketmq/store/index
messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h 6h 12h 1d 3d 15d
enablePropertyFilter=true
autoCreateTopicEnable=false`
3. Problem phenomenon：
A lot of flow control log information often occurs，By tracking the code, it is found that the synchronous replication between the master node and the slave node has timed out：
do sync transfer other node, wait return, but failed, topic
putMessage not in lock eclipse time(ms)=6194
log warn 1
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2 DESC: [TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: 209ms, size of queue: 56
log warn 2
2020-07-01 17:52:29 WARN SendMessageThread_8 - putMessage not in lock elapsed time(ms)=501, bodyLength=67
2020-07-01 17:52:29 WARN SendMessageThread_18 - putMessage not in lock elapsed time(ms)=501, bodyLength=67
2020-07-01 17:58:28 WARN SendMessageThread_3 - [NOTIFYME]putMessage in lock cost time(ms)=592, bodyLength=67 AppendMessageResult=AppendMessageResult{status=PUT_OK, wroteOffset=262888849194, wroteBytes=992, msgId='0A6F0C2D00002A9F0000003D35657F2A', storeTimestamp=1593597507430, logicsOffset=255311, pagecacheRT=592, msgNum=1}
2020-07-01 17:58:28 WARN SendMessageThread_19 - putMessage not in lock elapsed time(ms)=592, bodyLength=67
2020-07-01 17:58:28 WARN SendMessageThread_3 - putMessage not in lock elapsed time(ms)=593, bodyLength=67
2020-07-01 17:58:28 WARN SendMessageThread_16 - putMessage not in lock elapsed time(ms)=590, bodyLength=67
2020-07-01 17:58:28 WARN SendMessageThread_22 - putMessage not in lock elapsed time(ms)=589, bodyLength=67
4. How to troubleshoot and solve such problems？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] kayyellow edited a comment on issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
Posted by GitBox <gi...@apache.org>.
kayyellow edited a comment on issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132#issuecomment-654005753
> @kayyellow
> maybe brokerRole cause that
>
> if ASYNC_MASTER
> send to master broker, return send-result to client , async to slave broker
>
> if SYNC_MASTER and Property key isWaitStoreMsgOK == true (isWaitStoreMsgOK default value is true)
> send to master broker and sync to slave broker, return send-result to client (tps is very low)
>
> `public enum BrokerRole { ASYNC_MASTER, SYNC_MASTER, SLAVE; }`
OK， thx, I will try it.
##1515
According to the issue above， upgrade to 4.7.x seems can alse improve the performance of SYNC_MASTER. Should I change the brokerRole to ASYNC_MASTER, alse upgrade to 4.7.x to improve the performance？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] francisoliverlee commented on issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132#issuecomment-653779745
@kayyellow
maybe brokerRole cause that
if ASYNC_MASTER
send to master broker, return send-result to client , async to slave broker
if SYNC_MASTER and Property key isWaitStoreMsgOK == true
send to master broker and sync to slave broker, return send-result to client (tps is very low)
`public enum BrokerRole {
ASYNC_MASTER,
SYNC_MASTER,
SLAVE;
}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] kayyellow commented on issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
Posted by GitBox <gi...@apache.org>.
kayyellow commented on issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132#issuecomment-654005753
> @kayyellow
> maybe brokerRole cause that
>
> if ASYNC_MASTER
> send to master broker, return send-result to client , async to slave broker
>
> if SYNC_MASTER and Property key isWaitStoreMsgOK == true (isWaitStoreMsgOK default value is true)
> send to master broker and sync to slave broker, return send-result to client (tps is very low)
>
> `public enum BrokerRole { ASYNC_MASTER, SYNC_MASTER, SLAVE; }`
OK， thx, I will try it.
##1515
According to the above issue， upgrade to 4.7.x seems can alse improve the performance of SYNC_MASTER. Should I change the brokerRole to ASYNC_MASTER, alse upgrade to 4.7.x to improve the performance？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] francisoliverlee edited a comment on issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
Posted by GitBox <gi...@apache.org>.
francisoliverlee edited a comment on issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132#issuecomment-653779745
@kayyellow
maybe brokerRole cause that
if ASYNC_MASTER
send to master broker, return send-result to client , async to slave broker
if SYNC_MASTER and Property key isWaitStoreMsgOK == true (isWaitStoreMsgOK default value is true)
send to master broker and sync to slave broker, return send-result to client (tps is very low)
`public enum BrokerRole {
ASYNC_MASTER,
SYNC_MASTER,
SLAVE;
}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] vongosling closed issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
Posted by GitBox <gi...@apache.org>.
vongosling closed issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] francisoliverlee commented on issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
Posted by GitBox <gi...@apache.org>.
francisoliverlee commented on issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132#issuecomment-654530146
it depends on how reliable you want rmq is. see the "code"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] kayyellow removed a comment on issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
Posted by GitBox <gi...@apache.org>.
kayyellow removed a comment on issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132#issuecomment-654005387
#https://github.com/apache/rocketmq/issues/1515
According to the above issue， upgrade to 4.7.x seems can alse improve the performance of SYNC_MASTER. Should I change the brokerRole to ASYNC_MASTER, alse upgrade to 4.7.x to improve the performance？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] kayyellow commented on issue #2132: RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]
Posted by GitBox <gi...@apache.org>.
kayyellow commented on issue #2132:
URL: https://github.com/apache/rocketmq/issues/2132#issuecomment-654005387
#https://github.com/apache/rocketmq/issues/1515
According to the above issue， upgrade to 4.7.x seems can alse improve the performance of SYNC_MASTER. Should I change the brokerRole to ASYNC_MASTER, alse upgrade to 4.7.x to improve the performance？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org