[I] Locality impact from linux disk read ahead option [pinot]

["jasperjiaguo (via GitHub)" <gi...@apache.org>]

jasperjiaguo opened a new issue, #12166:
URL: https://github.com/apache/pinot/issues/12166

   Recently we have discovered the Pinot query latency can be impact by value of Linux's `read_ahead_kb`. Specifically we see a very high page fault count and severe tail latency shootup when the `read_ahead_kb` is set to a larger value in certain Linux distributions. As `read_ahead_kb` controls the read ahead during the access of mmap files, we think using larger value harms the queries having more random data access patterns. Theoretically it might benefit the opposite pattern but we have yet to see such a case. I think there are a few things that might worth doing:
   
   - In the short term we add this as a tip for Pinot admin in the OSS doc, so that it becomes a public knowledge
   - In the long term we may explore controlling this programmatically (like madvise in c), but it might be harder to do in Java
   - Revisit the mmap based segment cache
   
   Similar issues/analysis:
   https://smalldatum.blogspot.com/2014/05/the-impact-of-read-ahead-and-read-size.html
   https://github.com/elastic/elasticsearch/issues/27748
   
    
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org