[incubator-pinot] branch master updated: When initializing mutable dictionary, preserve 10% buffer for cardinality to reduce the chance of re-sizing the dictionary (#4363)

[ja...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
     new c768df8  When initializing mutable dictionary, preserve 10% buffer for cardinality to reduce the chance of re-sizing the dictionary (#4363)
c768df8 is described below

commit c768df89ef52e80df1595a98721fdf364f37bfad
Author: Xiaotian (Jackie) Jiang <17...@users.noreply.github.com>
AuthorDate: Mon Jul 29 20:25:30 2019 -0700

    When initializing mutable dictionary, preserve 10% buffer for cardinality to reduce the chance of re-sizing the dictionary (#4363)
    
    The current off-heap mutable dictionary is designed to hold all values in a single buffer
    If the buffer allocated is just enough for the cardinality estimation, there are quite high possibility that the dictionary needs to be expanded, which will impact performance
    In order to reduce the chance of re-sizing the dictionary, preserve 10% buffer for cardinality
---
 .../apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java   | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
index 457d8c7..e1334ae 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
@@ -44,7 +44,6 @@ import org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory;
 import org.apache.pinot.core.realtime.impl.dictionary.MutableDictionary;
 import org.apache.pinot.core.realtime.impl.dictionary.MutableDictionaryFactory;
 import org.apache.pinot.core.realtime.impl.invertedindex.RealtimeInvertedIndexReader;
-import org.apache.pinot.core.realtime.stream.StreamMessageMetadata;
 import org.apache.pinot.core.segment.creator.impl.V1Constants;
 import org.apache.pinot.core.segment.index.SegmentMetadataImpl;
 import org.apache.pinot.core.segment.index.data.source.ColumnDataSource;
@@ -163,10 +162,12 @@ public class MutableSegmentImpl implements MutableSegment {
         } else {
           dictionaryColumnSize = dataType.size();
         }
+        // NOTE: preserve 10% buffer for cardinality to reduce the chance of re-sizing the dictionary
+        int estimatedCardinality = (int) (_statsHistory.getEstimatedCardinality(column) * 1.1);
         String allocationContext = buildAllocationContext(_segmentName, column, V1Constants.Dict.FILE_EXTENSION);
         MutableDictionary dictionary = MutableDictionaryFactory
             .getMutableDictionary(dataType, _offHeap, _memoryManager, dictionaryColumnSize,
-                Math.min(_statsHistory.getEstimatedCardinality(column), _capacity), allocationContext);
+                Math.min(estimatedCardinality, _capacity), allocationContext);
         _dictionaryMap.put(column, dictionary);
 
         // Even though the column is defined as 'no-dictionary' in the config, we did create dictionary for consuming segment.


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org