You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by 国美金控投资有限公司-国美金控-信息技术中心-孔令超-13522338363 <ko...@gomefinance.com.cn> on 2019/02/26 07:38:30 UTC
Re:rocketmq问题排查
信息补充
----------------
All The Best!
孔令超
国美金融 /运维中心
Mobile:13522338363
E-Mail:konglingchao@gomefinance.com.cn
Address:北京市朝阳区霄云路26号鹏润大厦B座8层
Floor 19, Unit B, PengRun building, NO.26, XiaoYun Road, Chaoyang District, Beijing, P.R.C. 100125
---------- Origin message ----------
>From："国美金控投资有限公司-国美金控-信息技术中心-孔令超-13522338363" <ko...@gomefinance.com.cn>
>To：dev@rocketmq.apache.org
>Subject：rocketmq问题排查
>Date：2019-02-26 15:16:22
您好：
rocketmq 版本 V4_0_0_SNAPSHOT 架构3主3从
配置文件 master
namesrvAddr=10.162.0.126:9876;10.162.0.127:9876;10.162.0.128:9876
brokerClusterName=Online-znyx
brokerName=broker-b
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
listenPort=10110
storePathRootDir=/data/servers/apache-rocketmq-all/rocketmqlogs/store/
storePathCommitLog=/data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master
mapedFileSizeCommitLog=5242880
useReentrantLockWhenPutMessage=true
waitTimeMillsInSendQueue=600
sendMessageThreadPoolNums=256
配置文件slave
namesrvAddr=10.162.0.126:9876;10.162.0.127:9876;10.162.0.128:9876
brokerClusterName=Online-znyx
brokerName=broker-c
brokerId=1
deleteWhen=04
fileReservedTime=48
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
listenPort=10114
storePathRootDir=/data/servers/apache-rocketmq-all/rocketmqlogs/store/
storePathCommitLog=/data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/slave
mapedFileSizeCommitLog=5242880
useReentrantLockWhenPutMessage=true
waitTimeMillsInSendQueue=600
sendMessageThreadPoolNums=256
问题1
生产者 发送消息 ， rocketmq 返回成功，但mq未收到消息
报错日志
2019-02-26 13:39:16 WARN ReputMessageService - [BUG]logic queue order maybe wrong, expectLogicOffset: 2236400 currentLogicOffset: 2220900 Topic: event_bgdata_market QID: 0 Diff: 15500
2019-02-12 04:02:07 WARN WriteSocketService - Offset for /data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master/00000000000062914560 not matched. Request offset: 1584589995, index: 290, mappedFileSize: 5242880, mappedFiles count: 5
2019-02-12 04:02:07 WARN WriteSocketService - findMappedFileByOffset failure.
java.lang.ArrayIndexOutOfBoundsException: 290
at java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:368) ~[na:1.7.0_80]
at java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:377) ~[na:1.7.0_80]
at org.apache.rocketmq.store.MappedFileQueue.findMappedFileByOffset(MappedFileQueue.java:476) ~[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.CommitLog.getData(CommitLog.java:144) [rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.CommitLog.getData(CommitLog.java:139) [rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.DefaultMessageStore.getCommitLogData(DefaultMessageStore.java:696) [rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.ha.HAConnection$WriteSocketService.run(HAConnection.java:278) [rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
2019-02-12 04:02:07 WARN WriteSocketService - Offset for /data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master/00000000000062914560 not matched. Request offset: 1584589995, index: 290, mappedFileSize: 5242880, mappedFiles count: 5
烦请帮忙给点排查思路。
----------------
All The Best!
孔令超
国美金融 /运维中心
Mobile:13522338363
E-Mail:konglingchao@gomefinance.com.cn
Address:北京市朝阳区霄云路26号鹏润大厦B座8层
Floor 19, Unit B, PengRun building, NO.26, XiaoYun Road, Chaoyang District, Beijing, P.R.C. 100125
Re:Re:rocketmq问题排查
Posted by 叶茂伟 <ym...@126.com>.
生产者 发送消息 ， rocketmq 返回成功，但mq未收到消息
报错日志
2019-02-26 13:39:16 WARN ReputMessageService - [BUG]logic queue order maybe wrong, expectLogicOffset: 2236400 currentLogicOffset: 2220900 Topic: event_bgdata_market QID: 0 Diff: 15500
这个问题，产生的场景是：broker根据commitlog中的消息去构建consumequeue，构建到其中一条消息时，预期在consumequeue中的offset为2236400，但实际上consumequeue才构建到的offset为2220900。下标比预期的小了15500，一个consumequeue中的索引大小为20，这条问题消息之前的15500/20=775条消息为什么没构建出来？这里需要排查下。
2019-02-12 04:02:07 WARN WriteSocketService - Offset for /data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master/00000000000062914560 not matched. Request offset: 1584589995, index: 290, mappedFileSize: 5242880, mappedFiles count: 5
2019-02-12 04:02:07 WARN WriteSocketService - findMappedFileByOffset failure.
java.lang.ArrayIndexOutOfBoundsException: 290
at java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:368) ~[na:1.7.0_80]
at java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:377) ~[na:1.7.0_80]
at org.apache.rocketmq.store.MappedFileQueue.findMappedFileByOffset(MappedFileQueue.java:476) ~[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.CommitLog.getData(CommitLog.java:144) [rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.CommitLog.getData(CommitLog.java:139) [rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.DefaultMessageStore.getCommitLogData(DefaultMessageStore.java:696) [rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.ha.HAConnection$WriteSocketService.run(HAConnection.java:278) [rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
2019-02-12 04:02:07 WARN WriteSocketService - Offset for /data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master/00000000000062914560 not matched. Request offset: 1584589995, index: 290, mappedFileSize: 5242880, mappedFiles count: 5
这个问题日志里提示也比较明显了：每个commitlog的大小为5242880，当前系统中最小的commitlog为00000000000062914560（这个文件为第62914560/5242880=12个文件），当前commitlog目录下一共有5个文件，正常情况下后面4个文件应该为连续的，因此下标应该分别为：68157440、73400320、78643200、83886080。但是此时，系统要加载的下标1584589995，应该在第1584589995/5242880=302个文件中。由于目前系统中最小的文件为第12个，因此代码中拿到302-12=290这个下标去取mappedFile，这里就抛了一个数组下标越界。
这个问题需要你先去机器上确认一下，/data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master/下的commitlog文件是否有丢失，或者多了一个错误的文件。
在 2019-02-26 15:38:30，"国美金控投资有限公司-国美金控-信息技术中心-孔令超-13522338363" <ko...@gomefinance.com.cn> 写道：
信息补充
----------------
All The Best!
孔令超
国美金融 /运维中心
Mobile:13522338363
E-Mail:konglingchao@gomefinance.com.cn
Address:北京市朝阳区霄云路26号鹏润大厦B座8层
Floor 19, Unit B, PengRun building, NO.26, XiaoYun Road, Chaoyang District, Beijing, P.R.C. 100125
---------- Origin message ----------
>From："国美金控投资有限公司-国美金控-信息技术中心-孔令超-13522338363" <ko...@gomefinance.com.cn>
>To：dev@rocketmq.apache.org
>Subject：rocketmq问题排查
>Date：2019-02-26 15:16:22
您好：
rocketmq 版本 V4_0_0_SNAPSHOT 架构3主3从
配置文件 master
namesrvAddr=10.162.0.126:9876;10.162.0.127:9876;10.162.0.128:9876
brokerClusterName=Online-znyx
brokerName=broker-b
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
listenPort=10110
storePathRootDir=/data/servers/apache-rocketmq-all/rocketmqlogs/store/
storePathCommitLog=/data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master
mapedFileSizeCommitLog=5242880
useReentrantLockWhenPutMessage=true
waitTimeMillsInSendQueue=600
sendMessageThreadPoolNums=256
配置文件slave
namesrvAddr=10.162.0.126:9876;10.162.0.127:9876;10.162.0.128:9876
brokerClusterName=Online-znyx
brokerName=broker-c
brokerId=1
deleteWhen=04
fileReservedTime=48
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
listenPort=10114
storePathRootDir=/data/servers/apache-rocketmq-all/rocketmqlogs/store/
storePathCommitLog=/data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/slave
mapedFileSizeCommitLog=5242880
useReentrantLockWhenPutMessage=true
waitTimeMillsInSendQueue=600
sendMessageThreadPoolNums=256
问题1
生产者 发送消息 ， rocketmq 返回成功，但mq未收到消息
报错日志
2019-02-26 13:39:16 WARN ReputMessageService - [BUG]logic queue order maybe wrong, expectLogicOffset: 2236400 currentLogicOffset: 2220900 Topic: event_bgdata_market QID: 0 Diff: 15500
2019-02-12 04:02:07 WARN WriteSocketService - Offset for /data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master/00000000000062914560 not matched. Request offset: 1584589995, index: 290, mappedFileSize: 5242880, mappedFiles count: 5
2019-02-12 04:02:07 WARN WriteSocketService - findMappedFileByOffset failure.
java.lang.ArrayIndexOutOfBoundsException: 290
at java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:368) ~[na:1.7.0_80]
at java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:377) ~[na:1.7.0_80]
at org.apache.rocketmq.store.MappedFileQueue.findMappedFileByOffset(MappedFileQueue.java:476) ~[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.CommitLog.getData(CommitLog.java:144)[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.CommitLog.getData(CommitLog.java:139)[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.DefaultMessageStore.getCommitLogData(DefaultMessageStore.java:696)[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at org.apache.rocketmq.store.ha.HAConnection$WriteSocketService.run(HAConnection.java:278)[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
at java.lang.Thread.run(Thread.java:745)[na:1.7.0_80]
2019-02-12 04:02:07 WARN WriteSocketService - Offset for /data/servers/apache-rocketmq-all/rocketmqlogs/store/commitlog/master/00000000000062914560 not matched. Request offset: 1584589995, index: 290, mappedFileSize: 5242880, mappedFiles count: 5
烦请帮忙给点排查思路。
----------------
All The Best!
孔令超
国美金融 /运维中心
Mobile:13522338363
E-Mail:konglingchao@gomefinance.com.cn
Address:北京市朝阳区霄云路26号鹏润大厦B座8层
Floor 19, Unit B, PengRun building, NO.26, XiaoYun Road, Chaoyang District, Beijing, P.R.C. 100125