[incubator-pinot] branch master updated: Raise the threshold of using MMAP for inverted index creation from 100M to 2G (#4358)

[ja...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
     new 81f433f  Raise the threshold of using MMAP for inverted index creation from 100M to 2G (#4358)
81f433f is described below

commit 81f433fbfd24890f9d003d02aaf3881dc33487f8
Author: Xiaotian (Jackie) Jiang <17...@users.noreply.github.com>
AuthorDate: Mon Jun 24 17:57:37 2019 -0700

    Raise the threshold of using MMAP for inverted index creation from 100M to 2G (#4358)
    
    We have experienced very high GC stopping the thread time while generating huge inverted index with MMAP files
    Writing to MMAP files could potentially cause super long stopping the thread time and cause the server/minion lose ZK connection
    Raise the threshold to 2G (use direct memory if value buffer size is smaller than 2G) to solve the issue
    Generating extremely large inverted index (over 500M values) could still face the same issue, but we want to keep the threshold to prevent running out of direct memory
---
 .../segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
index c4e30b7..b18f5a1 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
@@ -52,8 +52,8 @@ import org.roaringbitmap.buffer.MutableRoaringBitmap;
  * <p>Based on the number of values we need to store, we use direct memory or MMap file to allocate the buffer.
  */
 public final class OffHeapBitmapInvertedIndexCreator implements InvertedIndexCreator {
-  // Use MMapBuffer if the buffer size is larger than 100MB
-  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 25_000_000;
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
 
   private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
   private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org