You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by "syncwt (via GitHub)" <gi...@apache.org> on 2023/02/07 06:14:39 UTC
[GitHub] [rocketmq] syncwt opened a new issue, #6005: Broker-a busy but broker-b is normal
syncwt opened a new issue, #6005:
URL: https://github.com/apache/rocketmq/issues/6005
**BUG REPORT**
1. Please describe the issue you observed:
From the monitor of broker, I found broker's tps/qps always fluctuate at 1:00 am to 6:00 am（not disk clear time[`deleteWhen`]）. And Always broker-a decline and broker-b rise. It seems broker-a is wrong at that time. It would cause broker-b's load too high then produce message failed in sometimes .I checked broker log and founds nothing error about that.
- What did you do (The steps to reproduce)?
It happens every day. I prepared to build a broker-c to reduce pressure for trouble time.
- What is expected to see?
I expected to everything well.
- What did you see instead?
2. Please tell us about your environment:
rocketmq version :4.8.0-bin-release ,deployed by private.
2m-2s-async cluster mode . And master node is 4C12G.
3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):
broker's conf as list:
```conf
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=0
deleteWhen=06
fileReservedTime=168
waitTimeMillsInSendQueue=1000
#transientStorePoolEnable=true
transferMsgByHeap=false
sendMessageThreadPoolNums=16
useReentrantLockWhenPutMessage=true
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
```
And broker-b busy log as list:
```log
[PCBUSY_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: 753ms, size of queue: 116
```
or
```log
[REJECTREQUEST]system busy, start flow control for a while
```
or
```log
[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: 1000ms, size of queue: 395
```
The broker's machine measures is all normal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] syncwt commented on issue #6005: Broker-a busy but broker-b is normal
Posted by "syncwt (via GitHub)" <gi...@apache.org>.
syncwt commented on issue #6005:
URL: https://github.com/apache/rocketmq/issues/6005#issuecomment-1434159483
Through a period of investigation and analysis, It seems the different performance of machines's disk cause the problem. We are ongoing ensure and analyse. Maybe change a disk for broker-a is a right way to solve the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] syncwt commented on issue #6005: Broker-a busy but broker-b is normal
Posted by "syncwt (via GitHub)" <gi...@apache.org>.
syncwt commented on issue #6005:
URL: https://github.com/apache/rocketmq/issues/6005#issuecomment-1480578272
The issue is be confirmed. Tencent cbs cluster's load is too high that cause disk‘s `io wait` fluctuated. Then broker start flow control. We solved the problem by migrating cloud disk instance to a stable cluster finally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] syncwt closed issue #6005: Broker-a busy but broker-b is normal
Posted by "syncwt (via GitHub)" <gi...@apache.org>.
syncwt closed issue #6005: Broker-a busy but broker-b is normal
URL: https://github.com/apache/rocketmq/issues/6005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org