[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #4358: Raise the threshold of using MMAP for inverted index creation from 100M to 2G

[GitBox <gi...@apache.org>]

Jackie-Jiang opened a new pull request #4358: Raise the threshold of using MMAP for inverted index creation from 100M to 2G
URL: https://github.com/apache/incubator-pinot/pull/4358
 
 
   We have experienced very high GC stopping the thread time while generating huge inverted index with MMAP files
   Writing to MMAP files could potentially cause super long stopping the thread time and cause the server/minion lose ZK connection
   Raise the threshold to 2G (use direct memory if value buffer size is smaller than 2G) to solve the issue
   Generating extremely large inverted index (over 500M values) could still face the same issue, but we want to keep the threshold to prevent running out of direct memory

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org