You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2021/09/13 08:55:47 UTC
[GitHub] [rocketmq] Cczzzz opened a new issue #3350: not in lock elapsed time(ms)={}, bodyLength
Cczzzz opened a new issue #3350:
URL: https://github.com/apache/rocketmq/issues/3350
在服务端总有莫名其妙的超时情况发生:
2021-09-13 01:44:35 WARN SendMessageThread_14 - putMessage not in lock elapsed time(ms)=980, bodyLength=22
2021-09-13 01:44:35 WARN SendMessageThread_1 - putMessage not in lock elapsed time(ms)=981, bodyLength=6864
2021-09-13 01:44:35 WARN SendMessageThread_9 - putMessage not in lock elapsed time(ms)=982, bodyLength=6786
但是并没有[NOTIFYME]putMessage in lock cost time(ms) ，说明没有写入page cache 耗时很高的情况。可能是主从同步重新问题？
场景：
> tps <1k
> 4.7.0 on dledger
>16c 32g
> 大概3天出现一次
今天发现 出现异常时有磁盘io读取和page in 的峰值，还有出现 meory page faults 的峰值。
这因为拉取历史消息发生缺页然后需要加载整个page cache 到内存中吗，这个行为会堵塞其他的写或者读吗。
日志中可以看见
Offset not matched. Request offset: 447420757556, firstOffset: 841813590016, lastOffset: 920196743168, mappedFileSize: 1073741824, mappedFiles count: 73
说明是一直有人缺拉取历史数据的。
读取历史数据会很严重的影响写入吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] Cczzzz commented on issue #3350: not in lock elapsed time(ms)={}, bodyLength
Posted by GitBox <gi...@apache.org>.
Cczzzz commented on issue #3350:
URL: https://github.com/apache/rocketmq/issues/3350#issuecomment-919653517
> 有很多可能。
>
> 拉历史数据对ssd io的占用非常大。
>
> 没有"putMessage in lock"不代表没有问题，可能是不够500ms，没有打出日志来，但是积少成多，造成queue堆积的厉害。
拉取历史消息触发缺页时是否会堵塞写入呢，linux 下有log记录这种情况吗，这个问题我查好几天了。
除了这个情况，还有可能是dledger 存储库的问题吗，4.7用的是0.1 而最新版是0.22
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] areyouok commented on issue #3350: not in lock elapsed time(ms)={}, bodyLength
Posted by GitBox <gi...@apache.org>.
areyouok commented on issue #3350:
URL: https://github.com/apache/rocketmq/issues/3350#issuecomment-918107805
有很多可能。
拉历史数据对ssd io的占用非常大。
没有"putMessage in lock"不代表没有问题，可能是不够500ms，没有打出日志来，但是积少成多，造成queue堆积的厉害。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org