[GitHub] [rocketmq] sysulincn opened a new issue #1489: rmq sometimes starts flow control under low tps because PageCacheRT overtime

[GitBox <gi...@apache.org>]

sysulincn opened a new issue #1489: rmq sometimes starts flow control under low tps because PageCacheRT overtime
URL: https://github.com/apache/rocketmq/issues/1489
 
 
   log file:
   <code><pre>
   WARN SendMessageThread_31 - [NOTIFYME]putMessage in lock cost time(ms)=1083, bodyLength=796 AppendMessageResult=AppendMessageResult{status=PUT_OK, wroteOffset=49503414542, wroteBytes=3446, msgId='0A8E5A11000007C30000000B86A2290E', storeTimestamp=1560741673222, logicsOffset=7043259, pagecacheRT=1083, msgNum=1}
   </pre></code>
   os:
   CentOS Linux release 7.4.1708 (Core) 
   uname:Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64
   libc:glibc 2.17 NPTL 2.17
   cpu: 48
   memory: 376.35GB
   there's one master broker(brokerId=10), one slave broker(brokerId=21) on the server,  they got different broker name.
   I've searched the issues, https://github.com/apache/rocketmq/issues/721 got the same symptom, and the issuer solved the problem by setting linux system var 
   <code><pre>sudo sysctl -w vm.min_free_kbytes=1000000</pre></code>
   but I belevied it is not the same issue in my case, because my server already have  vm.min_free_kbytes=4096000, much larger than that.
   after google for sometime, I now got a question about the following code in  os.sh
   <code><pre>
   sudo sysctl -w vm.dirty_background_ratio=50
   sudo sysctl -w vm.dirty_ratio=50
   sudo sysctl -w vm.dirty_writeback_centisecs=360000
   </pre></code>
   I doubt that the pagecachRT is high because the vm block the io to flush dirty pages into disk when the dirty pages get to 50% of the total memory, and the background procedure hasn't got enough time to flush dirty pages because the two vars have the same value.
   here's the question:
   1. How can I fix this pageCacheRT overtime problem, and why? 
   2. why set vm.dirty_background_ratio the same as vm.dirty_ratio?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services