You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@rocketmq.apache.org by hooligan <79...@qq.com> on 2018/07/19 06:08:44 UTC
回复：求助，RocketMQ broker01主机cpu异常飙高问题！
谢谢，我试试，但是另外一台broker02的主机，一样的配置文件，sendMessageThreadPoolNums也是24，写入和消费量相当，为何没有出现这种锁竞争情况呢？
------------------
hooligan
------------------ 原始邮件 ------------------
发件人: "duhengforever"<du...@gmail.com>;
发送时间: 2018年7月19日(星期四) 下午2:06
收件人: "users@rocketmq.apache.org"<us...@rocketmq.apache.org>;
抄送: "users"<us...@rocketmq.apache.org>;
主题: 回复：求助，RocketMQ broker01主机cpu异常飙高问题！
从日志来看，是因为大量SendMessageThread竞争锁导致CAS时产生CPU空转，尝试将sendMessageThreadPoolNums设置为5试下。
发自网易邮箱大师
在2018年07月19日 13:59，hooligan<79...@qq.com> 写道：
hi， all
现网运行的rocketmq-3.2.6集群，配置了两组broker：borker01和broker02，每组borker都是一主一备，都是24核、128G内存的机器，其中borker01主机的cpu特别高80%左右，boker02主机则只有20%左右，两台机器的生产消费流量差不多，broker01稍大一点，iowait也差不多，就是cpu使用率差异比较大，看broker.log日志，发现broker01的日志中有如下的WARN日志，请教一下大家这个问题要怎么解决，多谢
2018-07-19 13:54:15 INFO StoreStatsService - put_tps 10083.910572186434
2018-07-19 13:54:15 INFO StoreStatsService - get_found_tps 10138.794998105343
2018-07-19 13:54:15 INFO StoreStatsService - get_miss_tps 5010.140204622963
2018-07-19 13:54:15 INFO StoreStatsService - get_transfered_tps 17993.891625615764
2018-07-19 13:54:24 INFO FlushCommitLogService - how much disk fall behind memory, 9829
2018-07-19 13:54:24 WARN SendMessageThread_24 - putMessage in lock eclipse time(ms) 1757
2018-07-19 13:54:24 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms) 1757
2018-07-19 13:54:24 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms) 1759
2018-07-19 13:54:24 WARN SendMessageThread_9 - putMessage not in lock eclipse time(ms) 1761
2018-07-19 13:54:24 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms) 1761
2018-07-19 13:54:24 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms) 1763
2018-07-19 13:54:24 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms) 1765
2018-07-19 13:54:24 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms) 1769
2018-07-19 13:54:24 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms) 1771
2018-07-19 13:54:24 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms) 1771
2018-07-19 13:54:24 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms) 1773
2018-07-19 13:54:24 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms) 1773
2018-07-19 13:54:24 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1776
2018-07-19 13:54:24 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms) 1780
2018-07-19 13:54:24 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms) 1782
2018-07-19 13:54:24 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms) 1784
2018-07-19 13:54:24 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms) 1787
2018-07-19 13:54:24 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms) 1790
2018-07-19 13:54:24 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms) 1790
2018-07-19 13:54:24 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms) 1791
2018-07-19 13:54:24 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms) 1792
2018-07-19 13:54:38 WARN SendMessageThread_18 - putMessage in lock eclipse time(ms) 1826
2018-07-19 13:54:38 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms) 1826
2018-07-19 13:54:38 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms) 1836
2018-07-19 13:54:38 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms) 1836
2018-07-19 13:54:38 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms) 1838
2018-07-19 13:54:38 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms) 1839
2018-07-19 13:54:38 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms) 1839
2018-07-19 13:54:38 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms) 1842
2018-07-19 13:54:38 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms) 1843
2018-07-19 13:54:38 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms) 1846
2018-07-19 13:54:38 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms) 1847
2018-07-19 13:54:38 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms) 1850
2018-07-19 13:54:38 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms) 1853
2018-07-19 13:54:38 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms) 1852
2018-07-19 13:54:38 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms) 1853
2018-07-19 13:54:39 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms) 1854
2018-07-19 13:54:39 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms) 1854
2018-07-19 13:54:39 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms) 1857
2018-07-19 13:54:39 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms) 1858
2018-07-19 13:54:39 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms) 1856
2018-07-19 13:54:39 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms) 1859
iostat日志：
iostat -x 2
avg-cpu: %user %nice %system %iowait %steal %idle
9.69 0.00 5.53 0.02 0.00 84.76
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 252.63 0.93 233.58 55.05 2883.11 25.06 0.02 0.11 2.43 0.10 0.10 2.27
avg-cpu: %user %nice %system %iowait %steal %idle
48.03 0.00 23.91 0.13 0.00 27.93
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1124.00 0.00 1298.50 0.00 14552.00 22.41 0.15 0.12 0.00 0.12 0.11 14.60
avg-cpu: %user %nice %system %iowait %steal %idle
46.66 0.00 22.98 0.09 0.00 30.27
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1211.00 0.00 1379.00 0.00 15394.00 22.33 0.13 0.10 0.00 0.10 0.09 12.80
rocketmq配置文件：
brokerClusterName=MQ_sz
brokerName=broker-01
brokerId=0
deleteWhen=04
fileReservedTime=24
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/data/rocketmq/store
storePathCommitLog=/data/rocketmq/store/commitlog
autoCreateTopicEnable=false
autoCreateSubscriptionGroup=false
messageDelayLevel=1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 15s 20s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 15m 20m 30m 1h 2h 12h 24h 48h 72h
sendMessageThreadPoolNums=24
------------------
hooligan
回复：求助，RocketMQ broker01主机cpu异常飙高问题！
Posted by hooligan <79...@qq.com>.
今天将broker01机器的配置修改了，sendMessageThreadPoolNums=2，但是还是出现了这个错误，不过因为线程数减少了，错误日志量减少了
这里是不是要直接配置成sendMessageThreadPoolNums=1才能彻底解决这个问题
2018-07-25 18:01:12 INFO StoreStatsService - put_tps 7737.515093346543
2018-07-25 18:01:12 INFO StoreStatsService - get_found_tps 10701.80191337193
2018-07-25 18:01:12 INFO StoreStatsService - get_miss_tps 8204.371652373138
2018-07-25 18:01:12 INFO StoreStatsService - get_transfered_tps 13256.277284126443
2018-07-25 18:01:56 WARN SendMessageThread_2 - putMessage in lock eclipse time(ms) 1589
2018-07-25 18:01:56 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1589
2018-07-25 18:02:13 INFO StoreStatsService - put_tps 7250.234662317079
2018-07-25 18:02:13 INFO StoreStatsService - get_found_tps 9270.638743133242
2018-07-25 18:02:13 INFO StoreStatsService - get_miss_tps 7297.2440641974545
2018-07-25 18:02:13 INFO StoreStatsService - get_transfered_tps 12015.079939064735
2018-07-25 18:02:43 INFO FlushCommitLogService - how much disk fall behind memory, 5049
2018-07-25 18:02:57 WARN SendMessageThread_2 - putMessage in lock eclipse time(ms) 1676
2018-07-25 18:02:57 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1677
2018-07-25 18:03:13 INFO StoreStatsService - put_tps 7703.551625955342
2018-07-25 18:03:13 INFO StoreStatsService - get_found_tps 10123.033118537389
2018-07-25 18:03:13 INFO StoreStatsService - get_miss_tps 7582.721414656077
2018-07-25 18:03:13 INFO StoreStatsService - get_transfered_tps 13400.554473250413
------------------
hooligan
------------------ 原始邮件 ------------------
发件人: "A-Yu"<pl...@163.com>;
发送时间: 2018年7月19日(星期四) 下午3:44
收件人: "hooligan"<79...@qq.com>;
抄送: "users"<us...@rocketmq.apache.org>;
主题: 回复：求助，RocketMQ broker01主机cpu异常飙高问题！
从配置上看，为24核CPU，是否应该不存在锁问题？CAS是因为竞争网络发送导致的吗？
A-Yu
邮箱：plzy86@163.com
签名由 网易邮箱大师 定制
在2018年07月19日 14:08，hooligan 写道：
谢谢，我试试，但是另外一台broker02的主机，一样的配置文件，sendMessageThreadPoolNums也是24，写入和消费量相当，为何没有出现这种锁竞争情况呢？
------------------
hooligan
------------------ 原始邮件 ------------------
发件人: "duhengforever"<du...@gmail.com>;
发送时间: 2018年7月19日(星期四) 下午2:06
收件人: "users@rocketmq.apache.org"<us...@rocketmq.apache.org>;
抄送: "users"<us...@rocketmq.apache.org>;
主题: 回复：求助，RocketMQ broker01主机cpu异常飙高问题！
从日志来看，是因为大量SendMessageThread竞争锁导致CAS时产生CPU空转，尝试将sendMessageThreadPoolNums设置为5试下。
发自网易邮箱大师
在2018年07月19日 13:59，hooligan<79...@qq.com> 写道：
hi， all
现网运行的rocketmq-3.2.6集群，配置了两组broker：borker01和broker02，每组borker都是一主一备，都是24核、128G内存的机器，其中borker01主机的cpu特别高80%左右，boker02主机则只有20%左右，两台机器的生产消费流量差不多，broker01稍大一点，iowait也差不多，就是cpu使用率差异比较大，看broker.log日志，发现broker01的日志中有如下的WARN日志，请教一下大家这个问题要怎么解决，多谢
2018-07-19 13:54:15 INFO StoreStatsService - put_tps 10083.910572186434
2018-07-19 13:54:15 INFO StoreStatsService - get_found_tps 10138.794998105343
2018-07-19 13:54:15 INFO StoreStatsService - get_miss_tps 5010.140204622963
2018-07-19 13:54:15 INFO StoreStatsService - get_transfered_tps 17993.891625615764
2018-07-19 13:54:24 INFO FlushCommitLogService - how much disk fall behind memory, 9829
2018-07-19 13:54:24 WARN SendMessageThread_24 - putMessage in lock eclipse time(ms) 1757
2018-07-19 13:54:24 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms) 1757
2018-07-19 13:54:24 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms) 1759
2018-07-19 13:54:24 WARN SendMessageThread_9 - putMessage not in lock eclipse time(ms) 1761
2018-07-19 13:54:24 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms) 1761
2018-07-19 13:54:24 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms) 1763
2018-07-19 13:54:24 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms) 1765
2018-07-19 13:54:24 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms) 1769
2018-07-19 13:54:24 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms) 1771
2018-07-19 13:54:24 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms) 1771
2018-07-19 13:54:24 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms) 1773
2018-07-19 13:54:24 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms) 1773
2018-07-19 13:54:24 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1776
2018-07-19 13:54:24 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms) 1780
2018-07-19 13:54:24 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms) 1782
2018-07-19 13:54:24 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms) 1784
2018-07-19 13:54:24 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms) 1787
2018-07-19 13:54:24 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms) 1790
2018-07-19 13:54:24 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms) 1790
2018-07-19 13:54:24 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms) 1791
2018-07-19 13:54:24 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms) 1792
2018-07-19 13:54:38 WARN SendMessageThread_18 - putMessage in lock eclipse time(ms) 1826
2018-07-19 13:54:38 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms) 1826
2018-07-19 13:54:38 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms) 1836
2018-07-19 13:54:38 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms) 1836
2018-07-19 13:54:38 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms) 1838
2018-07-19 13:54:38 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms) 1839
2018-07-19 13:54:38 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms) 1839
2018-07-19 13:54:38 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms) 1842
2018-07-19 13:54:38 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms) 1843
2018-07-19 13:54:38 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms) 1846
2018-07-19 13:54:38 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms) 1847
2018-07-19 13:54:38 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms) 1850
2018-07-19 13:54:38 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms) 1853
2018-07-19 13:54:38 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms) 1852
2018-07-19 13:54:38 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms) 1853
2018-07-19 13:54:39 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms) 1854
2018-07-19 13:54:39 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms) 1854
2018-07-19 13:54:39 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms) 1857
2018-07-19 13:54:39 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms) 1858
2018-07-19 13:54:39 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms) 1856
2018-07-19 13:54:39 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms) 1859
iostat日志：
iostat -x 2
avg-cpu: %user %nice %system %iowait %steal %idle
9.69 0.00 5.53 0.02 0.00 84.76
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 252.63 0.93 233.58 55.05 2883.11 25.06 0.02 0.11 2.43 0.10 0.10 2.27
avg-cpu: %user %nice %system %iowait %steal %idle
48.03 0.00 23.91 0.13 0.00 27.93
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1124.00 0.00 1298.50 0.00 14552.00 22.41 0.15 0.12 0.00 0.12 0.11 14.60
avg-cpu: %user %nice %system %iowait %steal %idle
46.66 0.00 22.98 0.09 0.00 30.27
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1211.00 0.00 1379.00 0.00 15394.00 22.33 0.13 0.10 0.00 0.10 0.09 12.80
rocketmq配置文件：
brokerClusterName=MQ_sz
brokerName=broker-01
brokerId=0
deleteWhen=04
fileReservedTime=24
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/data/rocketmq/store
storePathCommitLog=/data/rocketmq/store/commitlog
autoCreateTopicEnable=false
autoCreateSubscriptionGroup=false
messageDelayLevel=1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 15s 20s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 15m 20m 30m 1h 2h 12h 24h 48h 72h
sendMessageThreadPoolNums=24
------------------
hooligan
回复：求助，RocketMQ broker01主机cpu异常飙高问题！
Posted by A-Yu <pl...@163.com>.
从配置上看，为24核CPU，是否应该不存在锁问题？CAS是因为竞争网络发送导致的吗？
| |
A-Yu
|
|
邮箱：plzy86@163.com
|
签名由 网易邮箱大师 定制
在2018年07月19日 14:08，hooligan 写道：
谢谢，我试试，但是另外一台broker02的主机，一样的配置文件，sendMessageThreadPoolNums也是24，写入和消费量相当，为何没有出现这种锁竞争情况呢？
------------------
hooligan
------------------ 原始邮件 ------------------
发件人: "duhengforever"<du...@gmail.com>;
发送时间: 2018年7月19日(星期四) 下午2:06
收件人: "users@rocketmq.apache.org"<us...@rocketmq.apache.org>;
抄送: "users"<us...@rocketmq.apache.org>;
主题: 回复：求助，RocketMQ broker01主机cpu异常飙高问题！
从日志来看，是因为大量SendMessageThread竞争锁导致CAS时产生CPU空转，尝试将sendMessageThreadPoolNums设置为5试下。
发自网易邮箱大师
在2018年07月19日 13:59，hooligan<79...@qq.com> 写道：
hi， all
现网运行的rocketmq-3.2.6集群，配置了两组broker：borker01和broker02，每组borker都是一主一备，都是24核、128G内存的机器，其中borker01主机的cpu特别高80%左右，boker02主机则只有20%左右，两台机器的生产消费流量差不多，broker01稍大一点，iowait也差不多，就是cpu使用率差异比较大，看broker.log日志，发现broker01的日志中有如下的WARN日志，请教一下大家这个问题要怎么解决，多谢
2018-07-19 13:54:15 INFO StoreStatsService - put_tps 10083.910572186434
2018-07-19 13:54:15 INFO StoreStatsService - get_found_tps 10138.794998105343
2018-07-19 13:54:15 INFO StoreStatsService - get_miss_tps 5010.140204622963
2018-07-19 13:54:15 INFO StoreStatsService - get_transfered_tps 17993.891625615764
2018-07-19 13:54:24 INFO FlushCommitLogService - how much disk fall behind memory, 9829
2018-07-19 13:54:24 WARN SendMessageThread_24 - putMessage in lock eclipse time(ms) 1757
2018-07-19 13:54:24 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms) 1757
2018-07-19 13:54:24 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms) 1759
2018-07-19 13:54:24 WARN SendMessageThread_9 - putMessage not in lock eclipse time(ms) 1761
2018-07-19 13:54:24 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms) 1761
2018-07-19 13:54:24 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms) 1763
2018-07-19 13:54:24 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms) 1765
2018-07-19 13:54:24 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms) 1769
2018-07-19 13:54:24 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms) 1771
2018-07-19 13:54:24 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms) 1771
2018-07-19 13:54:24 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms) 1773
2018-07-19 13:54:24 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms) 1773
2018-07-19 13:54:24 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1776
2018-07-19 13:54:24 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms) 1778
2018-07-19 13:54:24 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms) 1780
2018-07-19 13:54:24 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms) 1782
2018-07-19 13:54:24 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms) 1784
2018-07-19 13:54:24 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms) 1787
2018-07-19 13:54:24 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms) 1790
2018-07-19 13:54:24 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms) 1790
2018-07-19 13:54:24 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms) 1791
2018-07-19 13:54:24 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms) 1792
2018-07-19 13:54:38 WARN SendMessageThread_18 - putMessage in lock eclipse time(ms) 1826
2018-07-19 13:54:38 WARN SendMessageThread_18 - putMessage not in lock eclipse time(ms) 1826
2018-07-19 13:54:38 WARN SendMessageThread_24 - putMessage not in lock eclipse time(ms) 1836
2018-07-19 13:54:38 WARN SendMessageThread_5 - putMessage not in lock eclipse time(ms) 1836
2018-07-19 13:54:38 WARN SendMessageThread_23 - putMessage not in lock eclipse time(ms) 1838
2018-07-19 13:54:38 WARN SendMessageThread_10 - putMessage not in lock eclipse time(ms) 1839
2018-07-19 13:54:38 WARN SendMessageThread_12 - putMessage not in lock eclipse time(ms) 1839
2018-07-19 13:54:38 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms) 1842
2018-07-19 13:54:38 WARN SendMessageThread_8 - putMessage not in lock eclipse time(ms) 1843
2018-07-19 13:54:38 WARN SendMessageThread_20 - putMessage not in lock eclipse time(ms) 1846
2018-07-19 13:54:38 WARN SendMessageThread_22 - putMessage not in lock eclipse time(ms) 1847
2018-07-19 13:54:38 WARN SendMessageThread_17 - putMessage not in lock eclipse time(ms) 1850
2018-07-19 13:54:38 WARN SendMessageThread_4 - putMessage not in lock eclipse time(ms) 1853
2018-07-19 13:54:38 WARN SendMessageThread_21 - putMessage not in lock eclipse time(ms) 1852
2018-07-19 13:54:38 WARN SendMessageThread_7 - putMessage not in lock eclipse time(ms) 1853
2018-07-19 13:54:39 WARN SendMessageThread_15 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_13 - putMessage not in lock eclipse time(ms) 1849
2018-07-19 13:54:39 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms) 1854
2018-07-19 13:54:39 WARN SendMessageThread_14 - putMessage not in lock eclipse time(ms) 1854
2018-07-19 13:54:39 WARN SendMessageThread_16 - putMessage not in lock eclipse time(ms) 1857
2018-07-19 13:54:39 WARN SendMessageThread_6 - putMessage not in lock eclipse time(ms) 1858
2018-07-19 13:54:39 WARN SendMessageThread_11 - putMessage not in lock eclipse time(ms) 1856
2018-07-19 13:54:39 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms) 1859
iostat日志：
iostat -x 2
avg-cpu: %user %nice %system %iowait %steal %idle
9.69 0.00 5.53 0.02 0.00 84.76
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 252.63 0.93 233.58 55.05 2883.11 25.06 0.02 0.11 2.43 0.10 0.10 2.27
avg-cpu: %user %nice %system %iowait %steal %idle
48.03 0.00 23.91 0.13 0.00 27.93
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1124.00 0.00 1298.50 0.00 14552.00 22.41 0.15 0.12 0.00 0.12 0.11 14.60
avg-cpu: %user %nice %system %iowait %steal %idle
46.66 0.00 22.98 0.09 0.00 30.27
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 1211.00 0.00 1379.00 0.00 15394.00 22.33 0.13 0.10 0.00 0.10 0.09 12.80
rocketmq配置文件：
brokerClusterName=MQ_sz
brokerName=broker-01
brokerId=0
deleteWhen=04
fileReservedTime=24
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
storePathRootDir=/data/rocketmq/store
storePathCommitLog=/data/rocketmq/store/commitlog
autoCreateTopicEnable=false
autoCreateSubscriptionGroup=false
messageDelayLevel=1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 15s 20s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 15m 20m 30m 1h 2h 12h 24h 48h 72h
sendMessageThreadPoolNums=24
------------------
hooligan