You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by ja...@apache.org on 2019/06/24 18:28:42 UTC
[incubator-pinot] branch off_heap_threshold created (now 7f88a12)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a change to branch off_heap_threshold
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
at 7f88a12 Raise the threshold of using MMAP for inverted index creation from 100M to 2G
This branch includes the following new commits:
new 7f88a12 Raise the threshold of using MMAP for inverted index creation from 100M to 2G
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 01/01: Raise the threshold of using MMAP for
inverted index creation from 100M to 2G
Posted by ja...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch off_heap_threshold
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 7f88a12203d8b0966203bbcca19e9d6c69879923
Author: Jackie (Xiaotian) Jiang <xa...@linkedin.com>
AuthorDate: Mon Jun 24 11:23:09 2019 -0700
Raise the threshold of using MMAP for inverted index creation from 100M to 2G
We have experienced very high GC stopping the thread time while generating huge inverted index with MMAP files
Writing to MMAP files could potentially cause super long stopping the thread time and cause the server/minion lose ZK connection
Raise the threshold to 2G (use direct memory if value buffer size is smaller than 2G) to solve the issue
Generating extremely large inverted index (over 500M values) could still face the same issue, but we want to keep the threshold to prevent running out of direct memory
---
.../segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
index c4e30b7..b18f5a1 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/OffHeapBitmapInvertedIndexCreator.java
@@ -52,8 +52,8 @@ import org.roaringbitmap.buffer.MutableRoaringBitmap;
* <p>Based on the number of values we need to store, we use direct memory or MMap file to allocate the buffer.
*/
public final class OffHeapBitmapInvertedIndexCreator implements InvertedIndexCreator {
- // Use MMapBuffer if the buffer size is larger than 100MB
- private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 25_000_000;
+ // Use MMapBuffer if the value buffer size is larger than 2G
+ private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org