You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by ja...@apache.org on 2020/07/06 19:24:33 UTC
[incubator-pinot] branch master updated: Make DataSource
independent of query execution (not extend Operator) (#5625)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 2d53876 Make DataSource independent of query execution (not extend Operator) (#5625)
2d53876 is described below
commit 2d53876f669e34d8536461c52d5fe344ad1d1439
Author: Xiaotian (Jackie) Jiang <17...@users.noreply.github.com>
AuthorDate: Mon Jul 6 12:24:18 2020 -0700
Make DataSource independent of query execution (not extend Operator) (#5625)
Changes:
- Make `DataSource` independent of the query execution (not extend `Operator`).
- Add interfaces for forward index.
- Make `ForwardIndexReaderContext` close-able so that resources (e.g. off-heap buffer) can be released.
- In `ForwardIndexReader', always read with `ForwardIndexReaderContext` and close it when the reading is done to prevent resource leak.
- Remove `SingleValueSet` and `MultiValueSet` and change caller logic to directly use forward index and prevent resource leak.
- Make `DataFetcher` directly use forward index and handle the type conversion within the class. Similar to Dictionary convention, type conversion among INT, LONG, FLOAT, DOUBLE, STRING, and between STRING and BYTES are supported.
Motivation:
Currently `DataSource` is modeled as an `Operator`, where values are returned as a block complying with the Operator interface.
This is confusing because of the following reasons:
- The block contains all the documents in the segment, instead of a block of at most 10000 documents as in the `Projection` layer.
- The values are always fetched with their document ids, instead of fetched as a block. Currently `BlockValSet` interface has 2 sets of APIs because of this, which is confusing.
- The `BlockValSet` returned by the `DataSource` is not really a value set, but a value reader on top of the forward index.
- Extra `BlockMetadata` has to be maintained which can cause unexpected problems (e.g. the issue fixed in #5619)
TODO:
There are 2 remaining places where `ForwardIndexReaderContext` needs to be closed:
- `ScanBasedDocIdIterator`
- `DataFetcher`
New inheritance hierarchy for the forward index classes:
- `ForwardIndexReader`
- Dictionary-encoded unsorted SV: `FixedBitSVForwardIndexReader`
- Dictionary-encoded sorted SV: `SortedIndexReaderImpl`
- Dictionary-encoded MV: `FixedBitMVForwardIndexReader`
- Raw fixed-byte SV: `FixedByteChunkSVForwardIndexReader`
- Raw var-byte SV: `VarByteChunkSVForwardIndexReader`
- `MutableForwardIndex` extends `ForwardIndexReader`
- Fixed-byte SV (for both dictionary-encoded and raw): `FixedByteSVMutableForwardIndex`
- Var-byte SV: `VarByteSVMutableForwardIndex`
- Fixed-byte MV (for both dictionary-encoded and raw): `FixedByteMVMutableForwardIndex`
- `ForwardIndexCreator`
- Dictionary-encoded unsorted SV: `SingleValueUnsortedForwardIndexCreator`
- Dictionary-encoded sorted SV: `SingleValueSortedForwardIndexCreator`
- Dictionary-encoded MV: `MultiValueUnsortedForwardIndexCreator`
- Raw fixed-byte SV: `SingleValueFixedByteRawIndexCreator`
- Raw var-byte SV: `SingleValueVarByteRawIndexCreator`
---
.../apache/pinot/core/common/BaseBlockValSet.java | 185 ---------
.../org/apache/pinot/core/common/BlockValSet.java | 130 +-----
.../org/apache/pinot/core/common/DataFetcher.java | 445 ++++++++++++++++-----
.../org/apache/pinot/core/common/DataSource.java | 24 +-
.../readers/MultiplePinotSegmentRecordReader.java | 4 +-
.../data/readers/PinotSegmentColumnReader.java | 165 +++-----
.../data/readers/PinotSegmentRecordReader.java | 13 +-
.../core/data/readers/sort/PinotSegmentSorter.java | 2 +-
.../pinot/core/indexsegment/IndexSegmentUtils.java | 80 ----
.../indexsegment/immutable/ImmutableSegment.java | 4 +-
.../immutable/ImmutableSegmentImpl.java | 20 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 201 +++++-----
.../reader/BaseSingleColumnMultiValueReader.java | 64 ---
.../reader/BaseSingleColumnSingleValueReader.java | 102 -----
.../pinot/core/io/reader/DataFileReader.java | 30 --
.../apache/pinot/core/io/reader/ReaderContext.java | 23 --
.../io/reader/SingleColumnMultiValueReader.java | 81 ----
.../io/reader/SingleColumnSingleValueReader.java | 151 -------
.../io/reader/SingleValueMultiColumnReader.java | 156 --------
.../core/io/reader/impl/ChunkReaderContext.java | 53 ---
.../core/io/reader/impl/ConstantSVSortedIndex.java | 73 ----
.../io/reader/impl/UnSortedValueReaderContext.java | 26 --
.../reader/impl/v1/FixedBitSingleValueReader.java | 61 ---
.../impl/v1/FixedByteChunkSingleValueReader.java | 142 -------
.../BaseSingleColumnMultiValueReaderWriter.java | 117 ------
.../BaseSingleColumnSingleValueReaderWriter.java | 155 -------
.../BaseSingleValueMultiColumnReaderWriter.java | 132 ------
...ixedByteSingleValueMultiColumnReaderWriter.java | 17 +-
.../io/writer/SingleColumnMultiValueWriter.java | 78 ----
.../io/writer/SingleColumnSingleValueWriter.java | 77 ----
.../io/writer/SingleValueMultiColumnWriter.java | 70 ----
...ter.java => BaseChunkSVForwardIndexWriter.java} | 57 +--
...iter.java => FixedBitMVForwardIndexWriter.java} | 75 +---
.../writer/impl/FixedBitSVForwardIndexWriter.java | 58 +++
...ava => FixedByteChunkSVForwardIndexWriter.java} | 33 +-
....java => VarByteChunkSVForwardIndexWriter.java} | 21 +-
.../writer/impl/v1/FixedBitSingleValueWriter.java | 87 ----
.../pinot/core/minion/RawIndexConverter.java | 51 ++-
.../core/minion/segment/ReducerRecordReader.java | 4 +-
.../core/operator/blocks/BlockMetadataImpl.java | 81 ----
.../core/operator/blocks/MultiValueBlock.java | 68 ----
.../core/operator/blocks/SingleValueBlock.java | 72 ----
.../dociditerators/MVScanDocIdIterator.java | 17 +-
.../dociditerators/SVScanDocIdIterator.java | 68 ++--
.../core/operator/docidsets/MVScanDocIdSet.java | 6 +-
.../core/operator/docidsets/SVScanDocIdSet.java | 6 +-
.../operator/docvalsets/ProjectionBlockValSet.java | 6 +-
.../core/operator/docvalsets/SingleValueSet.java | 185 ---------
.../operator/docvalsets/TransformBlockValSet.java | 6 +-
.../operator/filter/ScanBasedFilterOperator.java | 14 +-
.../filter/SortedIndexBasedFilterOperator.java | 7 +-
.../converter/stats/RealtimeColumnStatistics.java | 24 +-
.../stats/RealtimeNoDictionaryColStatistics.java | 6 +-
.../dictionary/DoubleOffHeapMutableDictionary.java | 9 +-
.../dictionary/FloatOffHeapMutableDictionary.java | 9 +-
.../dictionary/IntOffHeapMutableDictionary.java | 9 +-
.../dictionary/LongOffHeapMutableDictionary.java | 9 +-
.../forward/FixedByteMVMutableForwardIndex.java} | 209 ++++------
.../forward/FixedByteSVMutableForwardIndex.java} | 171 ++++----
.../forward/VarByteSVMutableForwardIndex.java} | 91 ++---
.../invertedindex/RealtimeInvertedIndexReader.java | 7 -
.../RealtimeLuceneTextIndexReader.java | 9 +-
.../creator/BaseSingleValueRawIndexCreator.java | 59 ---
.../core/segment/creator/ForwardIndexCreator.java | 151 +++++++
.../creator/MultiValueForwardIndexCreator.java | 30 --
.../creator/SingleValueForwardIndexCreator.java | 30 --
.../creator/SingleValueRawIndexCreator.java | 84 ----
.../creator/impl/SegmentColumnarIndexCreator.java | 101 ++---
.../fwd/MultiValueUnsortedForwardIndexCreator.java | 34 +-
.../fwd/SingleValueFixedByteRawIndexCreator.java | 75 ++--
.../fwd/SingleValueSortedForwardIndexCreator.java | 28 +-
.../SingleValueUnsortedForwardIndexCreator.java | 34 +-
.../fwd/SingleValueVarByteRawIndexCreator.java | 62 +--
.../index/column/BaseVirtualColumnProvider.java | 3 +-
.../segment/index/column/ColumnIndexContainer.java | 8 +-
.../DefaultNullValueVirtualColumnProvider.java | 20 +-
.../index/column/PhysicalColumnIndexContainer.java | 43 +-
.../segment/index/datasource/BaseDataSource.java | 49 +--
.../index/datasource/ImmutableDataSource.java | 7 +-
.../index/datasource/MutableDataSource.java | 10 +-
.../defaultcolumn/BaseDefaultColumnHandler.java | 73 ++--
.../loader/invertedindex/InvertedIndexHandler.java | 28 +-
.../loader/invertedindex/RangeIndexHandler.java | 26 +-
.../loader/invertedindex/TextIndexHandler.java | 33 +-
.../index/readers/BitmapInvertedIndexReader.java | 7 -
.../segment/index/readers/ForwardIndexReader.java | 245 ++++++++++++
.../index/readers/ForwardIndexReaderContext.java} | 11 +-
.../segment/index/readers/InvertedIndexReader.java | 9 +-
.../segment/index/readers/MutableForwardIndex.java | 419 +++++++++++++++++++
.../segment/index/readers/RangeIndexReader.java | 7 -
.../index/readers/SortedIndexReader.java} | 36 +-
.../constant/ConstantMVForwardIndexReader.java} | 38 +-
.../constant/ConstantMVInvertedIndexReader.java} | 19 +-
.../constant/ConstantSortedIndexReader.java} | 38 +-
.../forward/BaseChunkSVForwardIndexReader.java} | 108 +++--
.../forward/FixedBitMVForwardIndexReader.java} | 96 ++---
.../forward/FixedBitSVForwardIndexReader.java | 70 ++++
.../FixedByteChunkSVForwardIndexReader.java | 90 +++++
.../forward/VarByteChunkSVForwardIndexReader.java} | 60 ++-
.../readers/sorted}/SortedIndexReaderImpl.java | 68 ++--
.../index/readers/text/LuceneTextIndexReader.java | 17 +-
.../virtualcolumn/DocIdVirtualColumnProvider.java | 77 +---
.../virtualcolumn/VirtualColumnIndexContainer.java | 16 +-
.../virtualcolumn/VirtualColumnProvider.java | 6 +-
.../startree/v2/builder/BaseSingleTreeBuilder.java | 66 ++-
.../v2/builder/OffHeapSingleTreeBuilder.java | 1 +
.../v2/builder/OnHeapSingleTreeBuilder.java | 6 +-
.../core/startree/v2/store/StarTreeDataSource.java | 8 +-
.../startree/v2/store/StarTreeLoaderUtils.java | 24 +-
.../apache/pinot/core/common/DataFetcherTest.java | 311 +++++++++-----
.../core/common/RealtimeNoDictionaryTest.java | 323 ---------------
.../mutable/MutableSegmentImplTest.java | 48 ++-
.../impl/dictionary/MultiValueDictionaryTest.java | 10 +-
.../segment/index/creator/RawIndexCreatorTest.java | 109 +++--
.../pinot/core/startree/v2/BaseStarTreeV2Test.java | 217 ++++++----
.../index/forward/FixedBitMVForwardIndexTest.java | 98 +++++
.../forward/FixedByteChunkSVForwardIndexTest.java | 272 +++++++++++++
.../SortedForwardIndexReaderTest.java | 29 +-
.../VarByteChunkSVForwardIndexTest.java} | 134 +++----
.../FixedByteMVMutableForwardIndexTest.java} | 76 ++--
.../FixedByteSVMutableForwardIndexTest.java} | 49 +--
.../mutable/VarByteSVMutableForwardIndexTest.java} | 17 +-
.../pinot/index/reader/FileReaderTestUtils.java | 45 ---
.../pinot/index/reader/FixedBitMultiValueTest.java | 201 ----------
.../FixedByteChunkSingleValueReaderWriteTest.java | 365 -----------------
...ByteSingleValueMultiColumnReaderWriterTest.java | 2 +-
...ixedByteWidthRowColForwardIndexWriterTest.java} | 2 +-
.../pinot/segments/v1/creator/IntArraysTest.java | 50 ++-
.../pinot/perf/BenchmarkOfflineIndexReader.java | 80 ++--
.../pinot/perf/BenchmarkPinotDataBitSet.java | 62 +--
.../pinot/perf/ForwardIndexWriterBenchmark.java | 8 +-
.../apache/pinot/tools/scan/query/Projection.java | 56 ++-
.../tools/scan/query/SegmentQueryProcessor.java | 68 ++--
.../converter/DictionaryToRawIndexConverter.java | 63 ++-
134 files changed, 3810 insertions(+), 5771 deletions(-)
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/BaseBlockValSet.java b/pinot-core/src/main/java/org/apache/pinot/core/common/BaseBlockValSet.java
deleted file mode 100644
index 44981e5..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/BaseBlockValSet.java
+++ /dev/null
@@ -1,185 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.common;
-
-/**
- * Abstract base class implementation for BlockValSet
- */
-public abstract class BaseBlockValSet implements BlockValSet {
-
- @Override
- public int getIntValue(int docId) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long getLongValue(int docId) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public float getFloatValue(int docId) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public double getDoubleValue(int docId) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public String getStringValue(int docId) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public byte[] getBytesValue(int docId) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getIntValues(int docId, int[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getLongValues(int docId, long[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getFloatValues(int docId, float[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getDoubleValues(int docId, double[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getStringValues(int docId, String[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void getDictionaryIds(int[] docIds, int length, int[] dictIdBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void getIntValues(int[] docIds, int length, int[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void getLongValues(int[] docIds, int length, long[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void getFloatValues(int[] docIds, int length, float[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void getDoubleValues(int[] docIds, int length, double[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void getStringValues(int[] docIds, int length, String[] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void getBytesValues(int[] docIds, int length, byte[][] valueBuffer) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int[] getDictionaryIdsSV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int[] getIntValuesSV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long[] getLongValuesSV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public float[] getFloatValuesSV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public double[] getDoubleValuesSV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public String[] getStringValuesSV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public byte[][] getBytesValuesSV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int[][] getDictionaryIdsMV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int[][] getIntValuesMV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long[][] getLongValuesMV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public float[][] getFloatValuesMV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public double[][] getDoubleValuesMV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public String[][] getStringValuesMV() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int[] getNumMVEntries() {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/BlockValSet.java b/pinot-core/src/main/java/org/apache/pinot/core/common/BlockValSet.java
index f8bc2ea..2a0f875 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/BlockValSet.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/common/BlockValSet.java
@@ -22,138 +22,20 @@ import org.apache.pinot.spi.data.FieldSpec.DataType;
/**
- * TODO: Split BlockValSet into multiple interfaces. The docId based APIs do not apply to Block concept.
+ * The {@code BlockValSet} contains a block of values for a column (Projection layer) or a transform expression
+ * (Transform layer).
*/
public interface BlockValSet {
- DataType getValueType();
-
- boolean isSingleValue();
-
- /**
- * DOCUMENT ID BASED APIs
- */
-
- /**
- * NOTE: The following single value read APIs do not handle the data type conversion for performance concern. Caller
- * should always call the API that matches the data type of the {@code BlockValSet}.
- */
-
- /**
- * Returns the INT type single-value at the given document id.
- * <p>NOTE: Dictionary id is handled as INT type.
- */
- int getIntValue(int docId);
-
- /**
- * Returns the LONG type single-value at the given document id.
- */
- long getLongValue(int docId);
-
/**
- * Returns the FLOAT type single-value at the given document id.
+ * Returns the data type of the values in the value set.
*/
- float getFloatValue(int docId);
-
- /**
- * Returns the DOUBLE type single-value at the given document id.
- */
- double getDoubleValue(int docId);
-
- /**
- * Returns the STRING type single-value at the given document id.
- */
- String getStringValue(int docId);
-
- /**
- * Returns the BYTES type single-value at the given document id.
- */
- byte[] getBytesValue(int docId);
-
- /**
- * Reads the INT type multi-value at the given document id into the value buffer and returns the number of values in
- * the multi-value entry.
- * <p>The passed in value buffer should be large enough to hold all the values of a multi-value entry.
- * <p>NOTE: Dictionary id is handled as INT type.
- */
- int getIntValues(int docId, int[] valueBuffer);
-
- /**
- * Reads the LONG type multi-value at the given document id into the value buffer and returns the number of values in
- * the multi-value entry.
- * <p>The passed in value buffer should be large enough to hold all the values of a multi-value entry.
- */
- int getLongValues(int docId, long[] valueBuffer);
-
- /**
- * Reads the FLOAT type multi-value at the given document id into the value buffer and returns the number of values
- * in the multi-value entry.
- * <p>The passed in value buffer should be large enough to hold all the values of a multi-value entry.
- */
- int getFloatValues(int docId, float[] valueBuffer);
-
- /**
- * Reads the DOUBLE type multi-value at the given document id into the value buffer and returns the number of values
- * in the multi-value entry.
- * <p>The passed in value buffer should be large enough to hold all the values of a multi-value entry.
- */
- int getDoubleValues(int docId, double[] valueBuffer);
-
- /**
- * Reads the STRING type multi-value at the given document id into the value buffer and returns the number of values
- * in the multi-value entry.
- * <p>The passed in value buffer should be large enough to hold all the values of a multi-value entry.
- */
- int getStringValues(int docId, String[] valueBuffer);
-
- /**
- * NOTE: The following batch value read APIs should be able to handle the data type conversion. Caller can call any
- * API regardless of the data type of the {@code BlockValSet}.
- * TODO: Consider letting the caller handle the data type conversion because for different use cases, we might need to
- * convert data type differently.
- */
-
- /**
- * Batch reads the dictionary ids at the given document ids of the given length into the dictionary id buffer.
- * <p>The passed in dictionary id buffer size should be larger than or equal to the length.
- */
- void getDictionaryIds(int[] docIds, int length, int[] dictIdBuffer);
-
- /**
- * Batch reads the INT type single-values at the given document ids of the given length into the value buffer.
- * <p>The passed in value buffer size should be larger than or equal to the length.
- */
- void getIntValues(int[] docIds, int length, int[] valueBuffer);
-
- /**
- * Batch reads the LONG type single-values at the given document ids of the given length into the value buffer.
- * <p>The passed in value buffer size should be larger than or equal to the length.
- */
- void getLongValues(int[] docIds, int length, long[] valueBuffer);
-
- /**
- * Batch reads the FLOAT type single-values at the given document ids of the given length into the value buffer.
- * <p>The passed in value buffer size should be larger than or equal to the length.
- */
- void getFloatValues(int[] docIds, int length, float[] valueBuffer);
-
- /**
- * Batch reads the DOUBLE type single-values at the given document ids of the given length into the value buffer.
- * <p>The passed in value buffer size should be larger than or equal to the length.
- */
- void getDoubleValues(int[] docIds, int length, double[] valueBuffer);
-
- /**
- * Batch reads the STRING type single-values at the given document ids of the given length into the value buffer.
- * <p>The passed in value buffer size should be larger than or equal to the length.
- */
- void getStringValues(int[] docIds, int length, String[] valueBuffer);
+ DataType getValueType();
/**
- * Batch reads the BYTES type single-values at the given document ids of the given length into the value buffer.
- * <p>The passed in value buffer size should be larger than or equal to the length.
+ * Returns {@code true} if the value set is for a single-value column, {@code false} otherwise.
*/
- void getBytesValues(int[] docIds, int length, byte[][] valueBuffer);
+ boolean isSingleValue();
/**
* SINGLE-VALUED COLUMN APIs
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java b/pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
index fc6799f..0e447ed 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/common/DataFetcher.java
@@ -18,31 +18,34 @@
*/
package org.apache.pinot.core.common;
+import java.io.Closeable;
+import java.io.IOException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
-import org.apache.pinot.core.operator.docvalsets.MultiValueSet;
-import org.apache.pinot.core.operator.docvalsets.SingleValueSet;
+import javax.annotation.Nullable;
import org.apache.pinot.core.plan.DocIdSetPlanNode;
import org.apache.pinot.core.segment.index.readers.Dictionary;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
+import org.apache.pinot.spi.utils.BytesUtils;
/**
- * DataFetcher is a higher level abstraction for data fetching. Given an index segment, DataFetcher can manage the
- * DataSource, Dictionary, BlockValSet and BlockValIterator for this segment, preventing redundant construction for
- * these instances. DataFetcher can be used by both selection, aggregation and group-by data fetching process, reducing
- * duplicate codes and garbage collection.
+ * DataFetcher is a higher level abstraction for data fetching. Given the DataSource, DataFetcher can manage the
+ * readers (ForwardIndexReader and Dictionary) for the column, preventing redundant construction for these instances.
+ * DataFetcher can be used by both selection, aggregation and group-by data fetching process, reducing duplicate codes
+ * and garbage collection.
*/
+@SuppressWarnings({"rawtypes", "unchecked"})
public class DataFetcher {
// Thread local (reusable) buffer for single-valued column dictionary Ids
private static final ThreadLocal<int[]> THREAD_LOCAL_DICT_IDS =
ThreadLocal.withInitial(() -> new int[DocIdSetPlanNode.MAX_DOC_PER_CALL]);
- private final Map<String, Dictionary> _dictionaryMap;
- // For single-value columns
- private final Map<String, SingleValueSet> _singleValueSetMap;
- // For multi-value columns
- private final Map<String, MultiValueSet> _multiValueSetMap;
+ // TODO: Figure out a way to close the reader context within the ColumnValueReader
+ // ChunkReaderContext should be closed explicitly to release the off-heap buffer
+ private final Map<String, ColumnValueReader> _columnValueReaderMap;
private final int[] _reusableMVDictIds;
/**
@@ -51,26 +54,19 @@ public class DataFetcher {
* @param dataSourceMap Map from column to data source
*/
public DataFetcher(Map<String, DataSource> dataSourceMap) {
- int numColumns = dataSourceMap.size();
- _dictionaryMap = new HashMap<>(numColumns);
- _singleValueSetMap = new HashMap<>(numColumns);
- _multiValueSetMap = new HashMap<>(numColumns);
-
+ _columnValueReaderMap = new HashMap<>();
int maxNumValuesPerMVEntry = 0;
for (Map.Entry<String, DataSource> entry : dataSourceMap.entrySet()) {
String column = entry.getKey();
DataSource dataSource = entry.getValue();
- _dictionaryMap.put(column, dataSource.getDictionary());
+ ColumnValueReader columnValueReader =
+ new ColumnValueReader(dataSource.getForwardIndex(), dataSource.getDictionary());
+ _columnValueReaderMap.put(column, columnValueReader);
DataSourceMetadata dataSourceMetadata = dataSource.getDataSourceMetadata();
- BlockValSet blockValueSet = dataSource.nextBlock().getBlockValueSet();
- if (dataSourceMetadata.isSingleValue()) {
- _singleValueSetMap.put(column, (SingleValueSet) blockValueSet);
- } else {
- _multiValueSetMap.put(column, (MultiValueSet) blockValueSet);
+ if (!dataSourceMetadata.isSingleValue()) {
maxNumValuesPerMVEntry = Math.max(maxNumValuesPerMVEntry, dataSourceMetadata.getMaxNumValuesPerMVEntry());
}
}
-
_reusableMVDictIds = new int[maxNumValuesPerMVEntry];
}
@@ -87,7 +83,7 @@ public class DataFetcher {
* @param outDictIds Buffer for output
*/
public void fetchDictIds(String column, int[] inDocIds, int length, int[] outDictIds) {
- _singleValueSetMap.get(column).getDictionaryIds(inDocIds, length, outDictIds);
+ _columnValueReaderMap.get(column).readDictIds(inDocIds, length, outDictIds);
}
/**
@@ -99,14 +95,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchIntValues(String column, int[] inDocIds, int length, int[] outValues) {
- Dictionary dictionary = _dictionaryMap.get(column);
- if (dictionary != null) {
- int[] dictIds = THREAD_LOCAL_DICT_IDS.get();
- fetchDictIds(column, inDocIds, length, dictIds);
- dictionary.readIntValues(dictIds, length, outValues);
- } else {
- _singleValueSetMap.get(column).getIntValues(inDocIds, length, outValues);
- }
+ _columnValueReaderMap.get(column).readIntValues(inDocIds, length, outValues);
}
/**
@@ -118,14 +107,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchLongValues(String column, int[] inDocIds, int length, long[] outValues) {
- Dictionary dictionary = _dictionaryMap.get(column);
- if (dictionary != null) {
- int[] dictIds = THREAD_LOCAL_DICT_IDS.get();
- fetchDictIds(column, inDocIds, length, dictIds);
- dictionary.readLongValues(dictIds, length, outValues);
- } else {
- _singleValueSetMap.get(column).getLongValues(inDocIds, length, outValues);
- }
+ _columnValueReaderMap.get(column).readLongValues(inDocIds, length, outValues);
}
/**
@@ -137,14 +119,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchFloatValues(String column, int[] inDocIds, int length, float[] outValues) {
- Dictionary dictionary = _dictionaryMap.get(column);
- if (dictionary != null) {
- int[] dictIds = THREAD_LOCAL_DICT_IDS.get();
- fetchDictIds(column, inDocIds, length, dictIds);
- dictionary.readFloatValues(dictIds, length, outValues);
- } else {
- _singleValueSetMap.get(column).getFloatValues(inDocIds, length, outValues);
- }
+ _columnValueReaderMap.get(column).readFloatValues(inDocIds, length, outValues);
}
/**
@@ -156,14 +131,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchDoubleValues(String column, int[] inDocIds, int length, double[] outValues) {
- Dictionary dictionary = _dictionaryMap.get(column);
- if (dictionary != null) {
- int[] dictIds = THREAD_LOCAL_DICT_IDS.get();
- fetchDictIds(column, inDocIds, length, dictIds);
- dictionary.readDoubleValues(dictIds, length, outValues);
- } else {
- _singleValueSetMap.get(column).getDoubleValues(inDocIds, length, outValues);
- }
+ _columnValueReaderMap.get(column).readDoubleValues(inDocIds, length, outValues);
}
/**
@@ -175,14 +143,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchStringValues(String column, int[] inDocIds, int length, String[] outValues) {
- Dictionary dictionary = _dictionaryMap.get(column);
- if (dictionary != null) {
- int[] dictIds = THREAD_LOCAL_DICT_IDS.get();
- fetchDictIds(column, inDocIds, length, dictIds);
- dictionary.readStringValues(dictIds, length, outValues);
- } else {
- _singleValueSetMap.get(column).getStringValues(inDocIds, length, outValues);
- }
+ _columnValueReaderMap.get(column).readStringValues(inDocIds, length, outValues);
}
/**
@@ -194,14 +155,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchBytesValues(String column, int[] inDocIds, int length, byte[][] outValues) {
- Dictionary dictionary = _dictionaryMap.get(column);
- if (dictionary != null) {
- int[] dictIds = THREAD_LOCAL_DICT_IDS.get();
- fetchDictIds(column, inDocIds, length, dictIds);
- dictionary.readBytesValues(dictIds, length, outValues);
- } else {
- _singleValueSetMap.get(column).getBytesValues(inDocIds, length, outValues);
- }
+ _columnValueReaderMap.get(column).readBytesValues(inDocIds, length, outValues);
}
/**
@@ -217,11 +171,7 @@ public class DataFetcher {
* @param outDictIds Buffer for output
*/
public void fetchDictIds(String column, int[] inDocIds, int length, int[][] outDictIds) {
- MultiValueSet multiValueSet = _multiValueSetMap.get(column);
- for (int i = 0; i < length; i++) {
- int numMultiValues = multiValueSet.getIntValues(inDocIds[i], _reusableMVDictIds);
- outDictIds[i] = Arrays.copyOfRange(_reusableMVDictIds, 0, numMultiValues);
- }
+ _columnValueReaderMap.get(column).readDictIdsMV(inDocIds, length, outDictIds);
}
/**
@@ -233,12 +183,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchIntValues(String column, int[] inDocIds, int length, int[][] outValues) {
- MultiValueSet multiValueSet = _multiValueSetMap.get(column);
- for (int i = 0; i < length; i++) {
- int numMultiValues = multiValueSet.getIntValues(inDocIds[i], _reusableMVDictIds);
- outValues[i] = new int[numMultiValues];
- _dictionaryMap.get(column).readIntValues(_reusableMVDictIds, numMultiValues, outValues[i]);
- }
+ _columnValueReaderMap.get(column).readIntValuesMV(inDocIds, length, outValues);
}
/**
@@ -250,12 +195,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchLongValues(String column, int[] inDocIds, int length, long[][] outValues) {
- MultiValueSet multiValueSet = _multiValueSetMap.get(column);
- for (int i = 0; i < length; i++) {
- int numMultiValues = multiValueSet.getIntValues(inDocIds[i], _reusableMVDictIds);
- outValues[i] = new long[numMultiValues];
- _dictionaryMap.get(column).readLongValues(_reusableMVDictIds, numMultiValues, outValues[i]);
- }
+ _columnValueReaderMap.get(column).readLongValuesMV(inDocIds, length, outValues);
}
/**
@@ -267,12 +207,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchFloatValues(String column, int[] inDocIds, int length, float[][] outValues) {
- MultiValueSet multiValueSet = _multiValueSetMap.get(column);
- for (int i = 0; i < length; i++) {
- int numMultiValues = multiValueSet.getIntValues(inDocIds[i], _reusableMVDictIds);
- outValues[i] = new float[numMultiValues];
- _dictionaryMap.get(column).readFloatValues(_reusableMVDictIds, numMultiValues, outValues[i]);
- }
+ _columnValueReaderMap.get(column).readFloatValuesMV(inDocIds, length, outValues);
}
/**
@@ -284,12 +219,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchDoubleValues(String column, int[] inDocIds, int length, double[][] outValues) {
- MultiValueSet multiValueSet = _multiValueSetMap.get(column);
- for (int i = 0; i < length; i++) {
- int numMultiValues = multiValueSet.getIntValues(inDocIds[i], _reusableMVDictIds);
- outValues[i] = new double[numMultiValues];
- _dictionaryMap.get(column).readDoubleValues(_reusableMVDictIds, numMultiValues, outValues[i]);
- }
+ _columnValueReaderMap.get(column).readDoubleValuesMV(inDocIds, length, outValues);
}
/**
@@ -301,12 +231,7 @@ public class DataFetcher {
* @param outValues Buffer for output
*/
public void fetchStringValues(String column, int[] inDocIds, int length, String[][] outValues) {
- MultiValueSet multiValueSet = _multiValueSetMap.get(column);
- for (int i = 0; i < length; i++) {
- int numMultiValues = multiValueSet.getIntValues(inDocIds[i], _reusableMVDictIds);
- outValues[i] = new String[numMultiValues];
- _dictionaryMap.get(column).readStringValues(_reusableMVDictIds, numMultiValues, outValues[i]);
- }
+ _columnValueReaderMap.get(column).readStringValuesMV(inDocIds, length, outValues);
}
/**
@@ -318,9 +243,309 @@ public class DataFetcher {
* @param outNumValues Buffer for output
*/
public void fetchNumValues(String column, int[] inDocIds, int length, int[] outNumValues) {
- MultiValueSet multiValueSet = _multiValueSetMap.get(column);
- for (int i = 0; i < length; i++) {
- outNumValues[i] = multiValueSet.getIntValues(inDocIds[i], _reusableMVDictIds);
+ _columnValueReaderMap.get(column).readNumValuesMV(inDocIds, length, outNumValues);
+ }
+
+ /**
+ * Helper class to read values for a column from forward index and dictionary. For raw (non-dictionary-encoded)
+ * forward index, similar to Dictionary, type conversion among INT, LONG, FLOAT, DOUBLE, STRING is supported; type
+ * conversion between STRING and BYTES via Hex encoding/decoding is supported.
+ */
+ private class ColumnValueReader implements Closeable {
+ final ForwardIndexReader _reader;
+ final ForwardIndexReaderContext _readerContext;
+ final Dictionary _dictionary;
+
+ ColumnValueReader(ForwardIndexReader reader, @Nullable Dictionary dictionary) {
+ _reader = reader;
+ _readerContext = reader.createContext();
+ _dictionary = dictionary;
+ }
+
+ void readDictIds(int[] docIds, int length, int[] dictIdBuffer) {
+ _reader.readDictIds(docIds, length, dictIdBuffer, _readerContext);
+ }
+
+ void readIntValues(int[] docIds, int length, int[] valueBuffer) {
+ if (_dictionary != null) {
+ int[] dictIdBuffer = THREAD_LOCAL_DICT_IDS.get();
+ _reader.readDictIds(docIds, length, dictIdBuffer, _readerContext);
+ _dictionary.readIntValues(dictIdBuffer, length, valueBuffer);
+ } else {
+ switch (_reader.getValueType()) {
+ case INT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getInt(docIds[i], _readerContext);
+ }
+ break;
+ case LONG:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = (int) _reader.getLong(docIds[i], _readerContext);
+ }
+ break;
+ case FLOAT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = (int) _reader.getFloat(docIds[i], _readerContext);
+ }
+ break;
+ case DOUBLE:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = (int) _reader.getDouble(docIds[i], _readerContext);
+ }
+ break;
+ case STRING:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = Integer.parseInt(_reader.getString(docIds[i], _readerContext));
+ }
+ break;
+ default:
+ throw new IllegalStateException();
+ }
+ }
+ }
+
+ void readLongValues(int[] docIds, int length, long[] valueBuffer) {
+ if (_dictionary != null) {
+ int[] dictIdBuffer = THREAD_LOCAL_DICT_IDS.get();
+ _reader.readDictIds(docIds, length, dictIdBuffer, _readerContext);
+ _dictionary.readLongValues(dictIdBuffer, length, valueBuffer);
+ } else {
+ switch (_reader.getValueType()) {
+ case INT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getInt(docIds[i], _readerContext);
+ }
+ break;
+ case LONG:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getLong(docIds[i], _readerContext);
+ }
+ break;
+ case FLOAT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = (long) _reader.getFloat(docIds[i], _readerContext);
+ }
+ break;
+ case DOUBLE:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = (long) _reader.getDouble(docIds[i], _readerContext);
+ }
+ break;
+ case STRING:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = Long.parseLong(_reader.getString(docIds[i], _readerContext));
+ }
+ break;
+ default:
+ throw new IllegalStateException();
+ }
+ }
+ }
+
+ void readFloatValues(int[] docIds, int length, float[] valueBuffer) {
+ if (_dictionary != null) {
+ int[] dictIdBuffer = THREAD_LOCAL_DICT_IDS.get();
+ _reader.readDictIds(docIds, length, dictIdBuffer, _readerContext);
+ _dictionary.readFloatValues(dictIdBuffer, length, valueBuffer);
+ } else {
+ switch (_reader.getValueType()) {
+ case INT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getInt(docIds[i], _readerContext);
+ }
+ break;
+ case LONG:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getLong(docIds[i], _readerContext);
+ }
+ break;
+ case FLOAT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getFloat(docIds[i], _readerContext);
+ }
+ break;
+ case DOUBLE:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = (float) _reader.getDouble(docIds[i], _readerContext);
+ }
+ break;
+ case STRING:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = Float.parseFloat(_reader.getString(docIds[i], _readerContext));
+ }
+ break;
+ default:
+ throw new IllegalStateException();
+ }
+ }
+ }
+
+ void readDoubleValues(int[] docIds, int length, double[] valueBuffer) {
+ if (_dictionary != null) {
+ int[] dictIdBuffer = THREAD_LOCAL_DICT_IDS.get();
+ _reader.readDictIds(docIds, length, dictIdBuffer, _readerContext);
+ _dictionary.readDoubleValues(dictIdBuffer, length, valueBuffer);
+ } else {
+ switch (_reader.getValueType()) {
+ case INT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getInt(docIds[i], _readerContext);
+ }
+ break;
+ case LONG:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getLong(docIds[i], _readerContext);
+ }
+ break;
+ case FLOAT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getFloat(docIds[i], _readerContext);
+ }
+ break;
+ case DOUBLE:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getDouble(docIds[i], _readerContext);
+ }
+ break;
+ case STRING:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = Double.parseDouble(_reader.getString(docIds[i], _readerContext));
+ }
+ break;
+ default:
+ throw new IllegalStateException();
+ }
+ }
+ }
+
+ void readStringValues(int[] docIds, int length, String[] valueBuffer) {
+ if (_dictionary != null) {
+ int[] dictIdBuffer = THREAD_LOCAL_DICT_IDS.get();
+ _reader.readDictIds(docIds, length, dictIdBuffer, _readerContext);
+ _dictionary.readStringValues(dictIdBuffer, length, valueBuffer);
+ } else {
+ switch (_reader.getValueType()) {
+ case INT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = Integer.toString(_reader.getInt(docIds[i], _readerContext));
+ }
+ break;
+ case LONG:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = Long.toString(_reader.getLong(docIds[i], _readerContext));
+ }
+ break;
+ case FLOAT:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = Float.toString(_reader.getFloat(docIds[i], _readerContext));
+ }
+ break;
+ case DOUBLE:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = Double.toString(_reader.getDouble(docIds[i], _readerContext));
+ }
+ break;
+ case STRING:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getString(docIds[i], _readerContext);
+ }
+ break;
+ case BYTES:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = BytesUtils.toHexString(_reader.getBytes(docIds[i], _readerContext));
+ }
+ break;
+ default:
+ throw new IllegalStateException();
+ }
+ }
+ }
+
+ void readBytesValues(int[] docIds, int length, byte[][] valueBuffer) {
+ if (_dictionary != null) {
+ int[] dictIdBuffer = THREAD_LOCAL_DICT_IDS.get();
+ _reader.readDictIds(docIds, length, dictIdBuffer, _readerContext);
+ _dictionary.readBytesValues(dictIdBuffer, length, valueBuffer);
+ } else {
+ switch (_reader.getValueType()) {
+ case STRING:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = BytesUtils.toBytes(_reader.getString(docIds[i], _readerContext));
+ }
+ break;
+ case BYTES:
+ for (int i = 0; i < length; i++) {
+ valueBuffer[i] = _reader.getBytes(docIds[i], _readerContext);
+ }
+ break;
+ default:
+ throw new IllegalStateException();
+ }
+ }
+ }
+
+ void readDictIdsMV(int[] docIds, int length, int[][] dictIdsBuffer) {
+ for (int i = 0; i < length; i++) {
+ int numValues = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, _readerContext);
+ dictIdsBuffer[i] = Arrays.copyOfRange(_reusableMVDictIds, 0, numValues);
+ }
+ }
+
+ void readIntValuesMV(int[] docIds, int length, int[][] valuesBuffer) {
+ for (int i = 0; i < length; i++) {
+ int numValues = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, _readerContext);
+ int[] values = new int[numValues];
+ _dictionary.readIntValues(_reusableMVDictIds, numValues, values);
+ valuesBuffer[i] = values;
+ }
+ }
+
+ void readLongValuesMV(int[] docIds, int length, long[][] valuesBuffer) {
+ for (int i = 0; i < length; i++) {
+ int numValues = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, _readerContext);
+ long[] values = new long[numValues];
+ _dictionary.readLongValues(_reusableMVDictIds, numValues, values);
+ valuesBuffer[i] = values;
+ }
+ }
+
+ void readFloatValuesMV(int[] docIds, int length, float[][] valuesBuffer) {
+ for (int i = 0; i < length; i++) {
+ int numValues = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, _readerContext);
+ float[] values = new float[numValues];
+ _dictionary.readFloatValues(_reusableMVDictIds, numValues, values);
+ valuesBuffer[i] = values;
+ }
+ }
+
+ void readDoubleValuesMV(int[] docIds, int length, double[][] valuesBuffer) {
+ for (int i = 0; i < length; i++) {
+ int numValues = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, _readerContext);
+ double[] values = new double[numValues];
+ _dictionary.readDoubleValues(_reusableMVDictIds, numValues, values);
+ valuesBuffer[i] = values;
+ }
+ }
+
+ void readStringValuesMV(int[] docIds, int length, String[][] valuesBuffer) {
+ for (int i = 0; i < length; i++) {
+ int numValues = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, _readerContext);
+ String[] values = new String[numValues];
+ _dictionary.readStringValues(_reusableMVDictIds, numValues, values);
+ valuesBuffer[i] = values;
+ }
+ }
+
+ public void readNumValuesMV(int[] docIds, int length, int[] numValuesBuffer) {
+ for (int i = 0; i < length; i++) {
+ numValuesBuffer[i] = _reader.getDictIdMV(docIds[i], _reusableMVDictIds, _readerContext);
+ }
+ }
+
+ @Override
+ public void close()
+ throws IOException {
+ _readerContext.close();
}
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/DataSource.java b/pinot-core/src/main/java/org/apache/pinot/core/common/DataSource.java
index 492c344..67a5124 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/DataSource.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/common/DataSource.java
@@ -19,56 +19,56 @@
package org.apache.pinot.core.common;
import javax.annotation.Nullable;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.operator.BaseOperator;
import org.apache.pinot.core.segment.index.readers.BloomFilterReader;
import org.apache.pinot.core.segment.index.readers.Dictionary;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
import org.apache.pinot.core.segment.index.readers.NullValueVectorReader;
/**
- * The {@code DataSource} class contains all the data and metadata for a column for query execution purpose.
+ * The {@code DataSource} contains all the indexes and metadata for a column for query execution purpose.
*/
-public abstract class DataSource extends BaseOperator {
+public interface DataSource {
/**
* Returns the metadata for the column.
*/
- public abstract DataSourceMetadata getDataSourceMetadata();
+ DataSourceMetadata getDataSourceMetadata();
/**
* Returns the forward index for the column. The forward index can be either dictionary-encoded or raw.
*/
- public abstract DataFileReader getForwardIndex();
+ ForwardIndexReader<?> getForwardIndex();
/**
* Returns the dictionary for the column if it is dictionary-encoded, or {@code null} if not.
*/
@Nullable
- public abstract Dictionary getDictionary();
+ Dictionary getDictionary();
/**
* Returns the inverted index for the column if exists, or {@code null} if not.
*/
@Nullable
- public abstract InvertedIndexReader getInvertedIndex();
+ InvertedIndexReader<?> getInvertedIndex();
/**
- * Returns the inverted index for the column if exists, or {@code null} if not.
+ * Returns the range index for the column if exists, or {@code null} if not.
+ * <p>TODO: Have a separate interface for range index.
*/
@Nullable
- public abstract InvertedIndexReader getRangeIndex();
+ InvertedIndexReader<?> getRangeIndex();
/**
* Returns the bloom filter for the column if exists, or {@code null} if not.
*/
@Nullable
- public abstract BloomFilterReader getBloomFilter();
+ BloomFilterReader getBloomFilter();
/**
* Returns null value vector for the column if exists, or {@code null} if not.
*/
@Nullable
- public abstract NullValueVectorReader getNullValueVector();
+ NullValueVectorReader getNullValueVector();
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/MultiplePinotSegmentRecordReader.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/MultiplePinotSegmentRecordReader.java
index a0b6fad..f40a58a 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/MultiplePinotSegmentRecordReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/MultiplePinotSegmentRecordReader.java
@@ -19,6 +19,7 @@
package org.apache.pinot.core.data.readers;
import java.io.File;
+import java.io.IOException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
@@ -187,7 +188,8 @@ public class MultiplePinotSegmentRecordReader implements RecordReader {
}
@Override
- public void close() {
+ public void close()
+ throws IOException {
for (PinotSegmentRecordReader recordReader : _pinotSegmentRecordReaders) {
recordReader.close();
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentColumnReader.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentColumnReader.java
index d9138de..9559b55 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentColumnReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentColumnReader.java
@@ -18,139 +18,82 @@
*/
package org.apache.pinot.core.data.readers;
+import java.io.Closeable;
+import java.io.IOException;
+import org.apache.pinot.core.common.DataSource;
import org.apache.pinot.core.indexsegment.immutable.ImmutableSegment;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.reader.SingleColumnMultiValueReader;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
-import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.core.segment.index.readers.Dictionary;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
-@SuppressWarnings("unchecked")
-public class PinotSegmentColumnReader {
+@SuppressWarnings({"rawtypes", "unchecked"})
+public class PinotSegmentColumnReader implements Closeable {
+ private final ForwardIndexReader _reader;
+ private final ForwardIndexReaderContext _readerContext;
private final Dictionary _dictionary;
- private final DataFileReader _reader;
- private final ReaderContext _readerContext;
- private final int[] _mvBuffer;
+ private final int[] _dictIdBuffer;
public PinotSegmentColumnReader(ImmutableSegment immutableSegment, String column) {
- _dictionary = immutableSegment.getDictionary(column);
- _reader = immutableSegment.getForwardIndex(column);
+ DataSource dataSource = immutableSegment.getDataSource(column);
+ _reader = dataSource.getForwardIndex();
_readerContext = _reader.createContext();
- SegmentMetadataImpl segmentMetadata = (SegmentMetadataImpl) immutableSegment.getSegmentMetadata();
- ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(column);
- if (columnMetadata.isSingleValue()) {
- _mvBuffer = null;
+ _dictionary = dataSource.getDictionary();
+ if (_reader.isSingleValue()) {
+ _dictIdBuffer = null;
} else {
- _mvBuffer = new int[columnMetadata.getMaxNumberOfMultiValues()];
+ _dictIdBuffer = new int[dataSource.getDataSourceMetadata().getMaxNumValuesPerMVEntry()];
}
}
- public Object readInt(int docId) {
- SingleColumnSingleValueReader svReader = (SingleColumnSingleValueReader) _reader;
- if (_dictionary != null) {
- int dictId = svReader.getInt(docId, _readerContext);
- return _dictionary.get(dictId);
- } else {
- return svReader.getInt(docId, _readerContext);
- }
- }
-
- public Object readLong(int docId) {
- SingleColumnSingleValueReader svReader = (SingleColumnSingleValueReader) _reader;
- if (_dictionary != null) {
- int dictId = svReader.getInt(docId, _readerContext);
- return _dictionary.get(dictId);
- } else {
- return svReader.getLong(docId, _readerContext);
- }
- }
-
- public Object readFloat(int docId) {
- SingleColumnSingleValueReader svReader = (SingleColumnSingleValueReader) _reader;
- if (_dictionary != null) {
- int dictId = svReader.getInt(docId, _readerContext);
- return _dictionary.get(dictId);
- } else {
- return svReader.getFloat(docId, _readerContext);
- }
- }
-
- public Object readDouble(int docId) {
- SingleColumnSingleValueReader svReader = (SingleColumnSingleValueReader) _reader;
- if (_dictionary != null) {
- int dictId = svReader.getInt(docId, _readerContext);
- return _dictionary.get(dictId);
- } else {
- return svReader.getDouble(docId, _readerContext);
- }
+ public boolean hasDictionary() {
+ return _dictionary != null;
}
- public Object readString(int docId) {
- SingleColumnSingleValueReader svReader = (SingleColumnSingleValueReader) _reader;
- if (_dictionary != null) {
- int dictId = svReader.getInt(docId, _readerContext);
- return _dictionary.get(dictId);
- } else {
- return svReader.getString(docId, _readerContext);
- }
+ public int getDictId(int docId) {
+ return _reader.getDictId(docId, _readerContext);
}
- public Object readBytes(int docId) {
- SingleColumnSingleValueReader svReader = (SingleColumnSingleValueReader) _reader;
+ public Object getValue(int docId) {
if (_dictionary != null) {
- int dictId = svReader.getInt(docId, _readerContext);
- return _dictionary.get(dictId);
+ if (_reader.isSingleValue()) {
+ return _dictionary.get(_reader.getDictId(docId, _readerContext));
+ } else {
+ int numValues = _reader.getDictIdMV(docId, _dictIdBuffer, _readerContext);
+ Object[] values = new Object[numValues];
+ for (int i = 0; i < numValues; i++) {
+ values[i] = _dictionary.get(_dictIdBuffer[i]);
+ }
+ return values;
+ }
} else {
- return svReader.getBytes(docId, _readerContext);
- }
- }
+ // NOTE: Only support single-value raw index
+ assert _reader.isSingleValue();
- public Object readSV(int docId, DataType dataType) {
- switch (dataType) {
- case INT:
- return readInt(docId);
- case LONG:
- return readLong(docId);
- case FLOAT:
- return readFloat(docId);
- case DOUBLE:
- return readDouble(docId);
- case STRING:
- return readString(docId);
- case BYTES:
- return readBytes(docId);
- default:
- throw new IllegalStateException("Unsupported data type: " + dataType);
+ switch (_reader.getValueType()) {
+ case INT:
+ return _reader.getInt(docId, _readerContext);
+ case LONG:
+ return _reader.getLong(docId, _readerContext);
+ case FLOAT:
+ return _reader.getFloat(docId, _readerContext);
+ case DOUBLE:
+ return _reader.getDouble(docId, _readerContext);
+ case STRING:
+ return _reader.getString(docId, _readerContext);
+ case BYTES:
+ return _reader.getBytes(docId, _readerContext);
+ default:
+ throw new IllegalStateException();
+ }
}
}
- public Object[] readMV(int docId) {
- SingleColumnMultiValueReader mvReader = (SingleColumnMultiValueReader) _reader;
- int numValues = mvReader.getIntArray(docId, _mvBuffer, _readerContext);
- Object[] values = new Object[numValues];
- for (int i = 0; i < numValues; i++) {
- values[i] = _dictionary.get(_mvBuffer[i]);
+ @Override
+ public void close()
+ throws IOException {
+ if (_readerContext != null) {
+ _readerContext.close();
}
- return values;
- }
-
- public int getDictionaryId(int docId) {
- if (_mvBuffer != null) {
- throw new IllegalStateException("Multi value column is not supported");
- }
- SingleColumnSingleValueReader svReader = (SingleColumnSingleValueReader) _reader;
- if (_dictionary != null) {
- return svReader.getInt(docId, _readerContext);
- } else {
- throw new IllegalStateException("No dictionary column is not supported");
- }
- }
-
- public boolean hasDictionary() {
- return _dictionary != null;
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentRecordReader.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentRecordReader.java
index a0a6221e..312cda7 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentRecordReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/PinotSegmentRecordReader.java
@@ -20,6 +20,7 @@ package org.apache.pinot.core.data.readers;
import com.google.common.base.Preconditions;
import java.io.File;
+import java.io.IOException;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
@@ -145,11 +146,7 @@ public class PinotSegmentRecordReader implements RecordReader {
private GenericRow getRecord(GenericRow reuse, int docId) {
for (FieldSpec fieldSpec : _schema.getAllFieldSpecs()) {
String fieldName = fieldSpec.getName();
- if (fieldSpec.isSingleValueField()) {
- reuse.putField(fieldName, _columnReaderMap.get(fieldName).readSV(docId, fieldSpec.getDataType()));
- } else {
- reuse.putField(fieldName, _columnReaderMap.get(fieldName).readMV(docId));
- }
+ reuse.putValue(fieldName, _columnReaderMap.get(fieldName).getValue(docId));
}
return reuse;
}
@@ -160,7 +157,11 @@ public class PinotSegmentRecordReader implements RecordReader {
}
@Override
- public void close() {
+ public void close()
+ throws IOException {
+ for (PinotSegmentColumnReader columnReader : _columnReaderMap.values()) {
+ columnReader.close();
+ }
_immutableSegment.destroy();
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/PinotSegmentSorter.java b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/PinotSegmentSorter.java
index 13ce154..ef07da3 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/PinotSegmentSorter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/readers/sort/PinotSegmentSorter.java
@@ -103,7 +103,7 @@ public class PinotSegmentSorter implements SegmentSorter {
}
// Compute the order
- compare = columnReader.getDictionaryId(docId1) - columnReader.getDictionaryId(docId2);
+ compare = columnReader.getDictId(docId1) - columnReader.getDictId(docId2);
if (compare != 0) {
return compare;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/IndexSegmentUtils.java b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/IndexSegmentUtils.java
deleted file mode 100644
index 0e11211..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/IndexSegmentUtils.java
+++ /dev/null
@@ -1,80 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.indexsegment;
-
-import javax.annotation.Nonnull;
-import javax.annotation.Nullable;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.SingleColumnMultiValueReader;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.core.segment.index.readers.Dictionary;
-
-
-public class IndexSegmentUtils {
- private IndexSegmentUtils() {
- }
-
- /**
- * Returns the value for the given document Id.
- *
- * @param docId Document Id
- * @param fieldSpec Field spec for the column
- * @param forwardIndex Forward index
- * @param dictionary Dictionary (apply to column with dictionary)
- * @param maxNumMultiValues Max number of multi-values for the column (apply to multi-valued column)
- * @return Value for the given document Id
- */
- public static Object getValue(int docId, @Nonnull FieldSpec fieldSpec, @Nonnull DataFileReader forwardIndex,
- @Nullable Dictionary dictionary, int maxNumMultiValues) {
- if (dictionary != null) {
- // Dictionary based
- if (fieldSpec.isSingleValueField()) {
- int dictId = ((SingleColumnSingleValueReader) forwardIndex).getInt(docId);
- return dictionary.get(dictId);
- } else {
- int[] dictIds = new int[maxNumMultiValues];
- int numValues = ((SingleColumnMultiValueReader) forwardIndex).getIntArray(docId, dictIds);
- Object[] value = new Object[numValues];
- for (int i = 0; i < numValues; i++) {
- value[i] = dictionary.get(dictIds[i]);
- }
- return value;
- }
- } else {
- // Raw index based
- // TODO: support multi-valued column
- SingleColumnSingleValueReader singleValueReader = (SingleColumnSingleValueReader) forwardIndex;
- switch (fieldSpec.getDataType()) {
- case INT:
- return singleValueReader.getInt(docId);
- case LONG:
- return singleValueReader.getLong(docId);
- case FLOAT:
- return singleValueReader.getFloat(docId);
- case DOUBLE:
- return singleValueReader.getDouble(docId);
- case STRING:
- return singleValueReader.getString(docId);
- default:
- throw new IllegalStateException();
- }
- }
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegment.java b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegment.java
index b7ae858..3198e9e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegment.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegment.java
@@ -19,7 +19,7 @@
package org.apache.pinot.core.indexsegment.immutable;
import org.apache.pinot.core.indexsegment.IndexSegment;
-import org.apache.pinot.core.io.reader.DataFileReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
import org.apache.pinot.core.segment.index.readers.Dictionary;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
@@ -40,7 +40,7 @@ public interface ImmutableSegment extends IndexSegment {
* @param column Column name
* @return Forward index for the given column
*/
- DataFileReader getForwardIndex(String column);
+ ForwardIndexReader getForwardIndex(String column);
/**
* Returns the inverted index for the given column.
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentImpl.java b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentImpl.java
index 5f139e0..42484cb 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentImpl.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentImpl.java
@@ -24,18 +24,15 @@ import java.util.Map;
import java.util.Set;
import javax.annotation.Nullable;
import org.apache.pinot.core.common.DataSource;
-import org.apache.pinot.core.indexsegment.IndexSegmentUtils;
-import org.apache.pinot.core.io.reader.DataFileReader;
import org.apache.pinot.core.segment.index.column.ColumnIndexContainer;
import org.apache.pinot.core.segment.index.datasource.ImmutableDataSource;
import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.core.segment.index.readers.Dictionary;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
import org.apache.pinot.core.segment.store.SegmentDirectory;
import org.apache.pinot.core.startree.v2.StarTreeV2;
import org.apache.pinot.core.startree.v2.store.StarTreeIndexContainer;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.data.readers.GenericRow;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -68,7 +65,7 @@ public class ImmutableSegmentImpl implements ImmutableSegment {
}
@Override
- public DataFileReader getForwardIndex(String column) {
+ public ForwardIndexReader getForwardIndex(String column) {
return _indexContainerMap.get(column).getForwardIndex();
}
@@ -138,16 +135,7 @@ public class ImmutableSegmentImpl implements ImmutableSegment {
@Override
public GenericRow getRecord(int docId, GenericRow reuse) {
- Schema schema = _segmentMetadata.getSchema();
- for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
- if (!fieldSpec.isVirtualColumn()) {
- String columnName = fieldSpec.getName();
- ColumnIndexContainer indexContainer = _indexContainerMap.get(columnName);
- reuse.putField(columnName, IndexSegmentUtils
- .getValue(docId, fieldSpec, indexContainer.getForwardIndex(), indexContainer.getDictionary(),
- _segmentMetadata.getColumnMetadataFor(columnName).getMaxNumberOfMultiValues()));
- }
- }
- return reuse;
+ // NOTE: Use PinotSegmentRecordReader to read immutable segment
+ throw new UnsupportedOperationException();
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
index c7ad7bd..020de0e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
@@ -36,18 +36,15 @@ import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.pinot.core.common.DataSource;
import org.apache.pinot.core.data.partition.PartitionFunction;
-import org.apache.pinot.core.indexsegment.IndexSegmentUtils;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.readerwriter.BaseSingleColumnSingleValueReaderWriter;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
-import org.apache.pinot.core.io.readerwriter.impl.FixedByteSingleColumnMultiValueReaderWriter;
-import org.apache.pinot.core.io.readerwriter.impl.FixedByteSingleColumnSingleValueReaderWriter;
-import org.apache.pinot.core.io.readerwriter.impl.VarByteSingleColumnSingleValueReaderWriter;
import org.apache.pinot.core.realtime.impl.RealtimeSegmentConfig;
import org.apache.pinot.core.realtime.impl.RealtimeSegmentStatsHistory;
import org.apache.pinot.core.realtime.impl.dictionary.BaseMutableDictionary;
import org.apache.pinot.core.realtime.impl.dictionary.BaseOffHeapMutableDictionary;
import org.apache.pinot.core.realtime.impl.dictionary.MutableDictionaryFactory;
+import org.apache.pinot.core.realtime.impl.forward.FixedByteMVMutableForwardIndex;
+import org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex;
+import org.apache.pinot.core.realtime.impl.forward.VarByteSVMutableForwardIndex;
import org.apache.pinot.core.realtime.impl.invertedindex.RealtimeInvertedIndexReader;
import org.apache.pinot.core.realtime.impl.invertedindex.RealtimeLuceneIndexRefreshState;
import org.apache.pinot.core.realtime.impl.invertedindex.RealtimeLuceneIndexRefreshState.RealtimeLuceneReaders;
@@ -60,6 +57,7 @@ import org.apache.pinot.core.segment.index.metadata.SegmentMetadata;
import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.core.segment.index.readers.BloomFilterReader;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
+import org.apache.pinot.core.segment.index.readers.MutableForwardIndex;
import org.apache.pinot.core.segment.index.readers.NullValueVectorReader;
import org.apache.pinot.core.segment.virtualcolumn.VirtualColumnContext;
import org.apache.pinot.core.segment.virtualcolumn.VirtualColumnProvider;
@@ -72,6 +70,7 @@ import org.apache.pinot.spi.config.table.ColumnPartitionConfig;
import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
import org.apache.pinot.spi.data.DimensionFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.apache.pinot.spi.data.MetricFieldSpec;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.data.readers.GenericRow;
@@ -112,7 +111,7 @@ public class MutableSegmentImpl implements MutableSegment {
// TODO: Keep one map to store all these info
private final Map<String, NumValuesInfo> _numValuesInfoMap = new HashMap<>();
private final Map<String, BaseMutableDictionary> _dictionaryMap = new HashMap<>();
- private final Map<String, DataFileReader> _indexReaderWriterMap = new HashMap<>();
+ private final Map<String, MutableForwardIndex> _forwardIndexMap = new HashMap<>();
private final Map<String, InvertedIndexReader> _invertedIndexMap = new HashMap<>();
private final Map<String, InvertedIndexReader> _rangeIndexMap = new HashMap<>();
private final Map<String, BloomFilterReader> _bloomFilterMap = new HashMap<>();
@@ -216,23 +215,35 @@ public class MutableSegmentImpl implements MutableSegment {
// Check whether to generate raw index for the column while consuming
// Only support generating raw index on single-value columns that do not have inverted index while
// consuming. After consumption completes and the segment is built, all single-value columns can have raw index
- FieldSpec.DataType dataType = fieldSpec.getDataType();
+ DataType dataType = fieldSpec.getDataType();
boolean isFixedWidthColumn = dataType.isFixedWidth();
- int forwardIndexColumnSize = -1;
if (isNoDictionaryColumn(noDictionaryColumns, invertedIndexColumns, fieldSpec, column)) {
- // no dictionary
- // each forward index entry will be equal to size of data for that row
- // For INT, LONG, FLOAT, DOUBLE it is equal to the number of fixed bytes used to store the value,
+ // No dictionary column (always single-valued)
+ assert fieldSpec.isSingleValueField();
+
+ String allocationContext =
+ buildAllocationContext(_segmentName, column, V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION);
if (isFixedWidthColumn) {
- forwardIndexColumnSize = dataType.size();
+ _forwardIndexMap.put(column,
+ new FixedByteSVMutableForwardIndex(false, dataType, _capacity, _memoryManager, allocationContext));
+ } else {
+ // RealtimeSegmentStatsHistory does not have the stats for no-dictionary columns from previous consuming
+ // segments
+ // TODO: Add support for updating RealtimeSegmentStatsHistory with average column value size for no dictionary
+ // columns as well
+ // TODO: Use the stats to get estimated average length
+ // Use a smaller capacity as opposed to segment flush size
+ int initialCapacity = Math.min(_capacity, NODICT_VARIABLE_WIDTH_ESTIMATED_NUMBER_OF_VALUES_DEFAULT);
+ _forwardIndexMap.put(column,
+ new VarByteSVMutableForwardIndex(dataType, _memoryManager, allocationContext, initialCapacity,
+ NODICT_VARIABLE_WIDTH_ESTIMATED_AVERAGE_VALUE_LENGTH_DEFAULT));
}
// Init min/max value map to avoid potential thread safety issue (expanding hashMap while reading).
_minValueMap.put(column, null);
_maxValueMap.put(column, null);
} else {
- // dictionary encoded index
- // each forward index entry will contain a 4 byte dictionary ID
- forwardIndexColumnSize = FieldSpec.DataType.INT.size();
+ // Dictionary-encoded column
+
int dictionaryColumnSize;
if (isFixedWidthColumn) {
dictionaryColumnSize = dataType.size();
@@ -241,57 +252,31 @@ public class MutableSegmentImpl implements MutableSegment {
}
// NOTE: preserve 10% buffer for cardinality to reduce the chance of re-sizing the dictionary
int estimatedCardinality = (int) (_statsHistory.getEstimatedCardinality(column) * 1.1);
- String allocationContext = buildAllocationContext(_segmentName, column, V1Constants.Dict.FILE_EXTENSION);
- BaseMutableDictionary dictionary = MutableDictionaryFactory
+ String dictionaryAllocationContext =
+ buildAllocationContext(_segmentName, column, V1Constants.Dict.FILE_EXTENSION);
+ _dictionaryMap.put(column, MutableDictionaryFactory
.getMutableDictionary(dataType, _offHeap, _memoryManager, dictionaryColumnSize,
- Math.min(estimatedCardinality, _capacity), allocationContext);
- _dictionaryMap.put(column, dictionary);
-
- // Even though the column is defined as 'no-dictionary' in the config, we did create dictionary for consuming segment.
- noDictionaryColumns.remove(column);
- }
-
- DataFileReader indexReaderWriter;
+ Math.min(estimatedCardinality, _capacity), dictionaryAllocationContext));
- // create forward index reader/writer
- if (forwardIndexColumnSize == -1) {
- // for STRING/BYTES SV column, we support raw index in consuming segments
- // RealtimeSegmentStatsHistory does not have the stats for no-dictionary columns
- // from previous consuming segments
- // TODO: Add support for updating RealtimeSegmentStatsHistory with average column value size for no dictionary columns as well
- // TODO: Use the stats to get estimated average length
- // Use a smaller capacity as opposed to segment flush size
- int initialCapacity = Math.min(_capacity, NODICT_VARIABLE_WIDTH_ESTIMATED_NUMBER_OF_VALUES_DEFAULT);
- String allocationContext =
- buildAllocationContext(_segmentName, column, V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
- indexReaderWriter =
- new VarByteSingleColumnSingleValueReaderWriter(_memoryManager, allocationContext, initialCapacity,
- NODICT_VARIABLE_WIDTH_ESTIMATED_AVERAGE_VALUE_LENGTH_DEFAULT);
- } else {
- // two possible cases can lead here:
- // (1) dictionary encoded forward index
- // (2) raw forward index for fixed width types -- INT, LONG, FLOAT, DOUBLE
if (fieldSpec.isSingleValueField()) {
- // SV column -- both dictionary encoded and raw index are supported on SV
- // columns for both fixed and variable width types
+ // Single-value dictionary-encoded forward index
String allocationContext = buildAllocationContext(_segmentName, column,
V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
- indexReaderWriter =
- new FixedByteSingleColumnSingleValueReaderWriter(_capacity, forwardIndexColumnSize, _memoryManager,
- allocationContext);
+ _forwardIndexMap.put(column,
+ new FixedByteSVMutableForwardIndex(true, DataType.INT, _capacity, _memoryManager, allocationContext));
} else {
- // MV column -- only dictionary encoded index is supported on MV columns
- // for both fixed and variable width types
- // TODO: Start with a smaller capacity on FixedByteSingleColumnMultiValueReaderWriter and let it expand
+ // Multi-value dictionary-encoded forward index
String allocationContext = buildAllocationContext(_segmentName, column,
V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION);
- indexReaderWriter =
- new FixedByteSingleColumnMultiValueReaderWriter(MAX_MULTI_VALUES_PER_ROW, avgNumMultiValues, _capacity,
- forwardIndexColumnSize, _memoryManager, allocationContext);
+ // TODO: Start with a smaller capacity on FixedByteMVForwardIndexReaderWriter and let it expand
+ _forwardIndexMap.put(column,
+ new FixedByteMVMutableForwardIndex(MAX_MULTI_VALUES_PER_ROW, avgNumMultiValues, _capacity, Integer.BYTES,
+ _memoryManager, allocationContext));
}
- }
- _indexReaderWriterMap.put(column, indexReaderWriter);
+ // Even though the column is defined as 'no-dictionary' in the config, we did create dictionary for consuming segment.
+ noDictionaryColumns.remove(column);
+ }
if (invertedIndexColumns.contains(column)) {
_invertedIndexMap.put(column, new RealtimeInvertedIndexReader());
@@ -333,7 +318,7 @@ public class MutableSegmentImpl implements MutableSegment {
*/
private boolean isNoDictionaryColumn(Set<String> noDictionaryColumns, Set<String> invertedIndexColumns,
FieldSpec fieldSpec, String column) {
- FieldSpec.DataType dataType = fieldSpec.getDataType();
+ DataType dataType = fieldSpec.getDataType();
if (noDictionaryColumns.contains(column)) {
// Earlier we didn't support noDict in consuming segments for STRING and BYTES columns.
// So even if the user had the column in noDictionaryColumns set in table config, we still
@@ -345,10 +330,11 @@ public class MutableSegmentImpl implements MutableSegment {
// of noDict on STRING/BYTES. Without metrics aggregation, memory pressure increases.
// So to continue aggregating metrics for such cases, we will create dictionary even
// if the column is part of noDictionary set from table config
- if (fieldSpec instanceof DimensionFieldSpec && _aggregateMetrics && (dataType == FieldSpec.DataType.STRING ||
- dataType == FieldSpec.DataType.BYTES)) {
- _logger.info("Aggregate metrics is enabled. Will create dictionary in consuming segment for column {} of type {}",
- column, dataType.toString());
+ if (fieldSpec instanceof DimensionFieldSpec && _aggregateMetrics && (dataType == DataType.STRING
+ || dataType == DataType.BYTES)) {
+ _logger
+ .info("Aggregate metrics is enabled. Will create dictionary in consuming segment for column {} of type {}",
+ column, dataType.toString());
return false;
}
// So don't create dictionary if the column is member of noDictionary, is single-value
@@ -513,33 +499,32 @@ public class MutableSegmentImpl implements MutableSegment {
NumValuesInfo numValuesInfo = _numValuesInfoMap.get(column);
if (fieldSpec.isSingleValueField()) {
// SV column
- BaseSingleColumnSingleValueReaderWriter indexReaderWriter =
- (BaseSingleColumnSingleValueReaderWriter) _indexReaderWriterMap.get(column);
+ MutableForwardIndex mutableForwardIndex = _forwardIndexMap.get(column);
Integer dictId = (Integer) dictIdMap.get(column);
if (dictId != null) {
// SV Column with dictionary
- indexReaderWriter.setInt(docId, dictId);
+ mutableForwardIndex.setDictId(docId, dictId);
} else {
// No-dictionary SV column
- FieldSpec.DataType dataType = fieldSpec.getDataType();
+ DataType dataType = fieldSpec.getDataType();
switch (dataType) {
case INT:
- indexReaderWriter.setInt(docId, (Integer) value);
+ mutableForwardIndex.setInt(docId, (Integer) value);
break;
case LONG:
- indexReaderWriter.setLong(docId, (Long) value);
+ mutableForwardIndex.setLong(docId, (Long) value);
break;
case FLOAT:
- indexReaderWriter.setFloat(docId, (Float) value);
+ mutableForwardIndex.setFloat(docId, (Float) value);
break;
case DOUBLE:
- indexReaderWriter.setDouble(docId, (Double) value);
+ mutableForwardIndex.setDouble(docId, (Double) value);
break;
case STRING:
- indexReaderWriter.setString(docId, (String) value);
+ mutableForwardIndex.setString(docId, (String) value);
break;
case BYTES:
- indexReaderWriter.setBytes(docId, (byte[]) value);
+ mutableForwardIndex.setBytes(docId, (byte[]) value);
break;
default:
throw new UnsupportedOperationException(
@@ -551,7 +536,7 @@ public class MutableSegmentImpl implements MutableSegment {
} else {
// MV column: always dictionary encoded
int[] dictIds = (int[]) dictIdMap.get(column);
- ((FixedByteSingleColumnMultiValueReaderWriter) _indexReaderWriterMap.get(column)).setIntArray(docId, dictIds);
+ _forwardIndexMap.get(column).setDictIdMV(docId, dictIds);
numValuesInfo.updateMVEntry(dictIds.length);
}
@@ -621,22 +606,20 @@ public class MutableSegmentImpl implements MutableSegment {
for (MetricFieldSpec metricFieldSpec : _physicalMetricFieldSpecs) {
String column = metricFieldSpec.getName();
Object value = row.getValue(column);
- FixedByteSingleColumnSingleValueReaderWriter indexReaderWriter =
- (FixedByteSingleColumnSingleValueReaderWriter) _indexReaderWriterMap.get(column);
-
- FieldSpec.DataType dataType = metricFieldSpec.getDataType();
+ MutableForwardIndex mutableForwardIndex = _forwardIndexMap.get(column);
+ DataType dataType = metricFieldSpec.getDataType();
switch (dataType) {
case INT:
- indexReaderWriter.setInt(docId, (Integer) value + indexReaderWriter.getInt(docId));
+ mutableForwardIndex.setInt(docId, (Integer) value + mutableForwardIndex.getInt(docId));
break;
case LONG:
- indexReaderWriter.setLong(docId, (Long) value + indexReaderWriter.getLong(docId));
+ mutableForwardIndex.setLong(docId, (Long) value + mutableForwardIndex.getLong(docId));
break;
case FLOAT:
- indexReaderWriter.setFloat(docId, (Float) value + indexReaderWriter.getFloat(docId));
+ mutableForwardIndex.setFloat(docId, (Float) value + mutableForwardIndex.getFloat(docId));
break;
case DOUBLE:
- indexReaderWriter.setDouble(docId, (Double) value + indexReaderWriter.getDouble(docId));
+ mutableForwardIndex.setDouble(docId, (Double) value + mutableForwardIndex.getDouble(docId));
break;
default:
throw new UnsupportedOperationException(
@@ -701,7 +684,7 @@ public class MutableSegmentImpl implements MutableSegment {
partitionId = _partitionId;
}
NumValuesInfo numValuesInfo = _numValuesInfoMap.get(column);
- DataFileReader forwardIndex = _indexReaderWriterMap.get(column);
+ MutableForwardIndex forwardIndex = _forwardIndexMap.get(column);
BaseMutableDictionary dictionary = _dictionaryMap.get(column);
InvertedIndexReader invertedIndex = _invertedIndexMap.get(column);
InvertedIndexReader rangeIndex = _rangeIndexMap.get(column);
@@ -710,8 +693,8 @@ public class MutableSegmentImpl implements MutableSegment {
Comparable maxValue = getMaxVal(column);
RealtimeNullValueVectorReaderWriter nullValueVector = _nullValueVectorMap.get(column);
return new MutableDataSource(fieldSpec, _numDocsIndexed, numValuesInfo.getNumValues(),
- numValuesInfo.getMaxNumValuesPerMVEntry(), partitionFunction, partitionId, minValue, maxValue,
- forwardIndex, dictionary, invertedIndex, rangeIndex, bloomFilter, nullValueVector);
+ numValuesInfo.getMaxNumValuesPerMVEntry(), partitionFunction, partitionId, minValue, maxValue, forwardIndex,
+ dictionary, invertedIndex, rangeIndex, bloomFilter, nullValueVector);
}
}
@@ -729,9 +712,8 @@ public class MutableSegmentImpl implements MutableSegment {
public GenericRow getRecord(int docId, GenericRow reuse) {
for (FieldSpec fieldSpec : _physicalFieldSpecs) {
String column = fieldSpec.getName();
- Object value = IndexSegmentUtils
- .getValue(docId, fieldSpec, _indexReaderWriterMap.get(column), _dictionaryMap.get(column),
- _numValuesInfoMap.get(column).getMaxNumValuesPerMVEntry());
+ Object value = getValue(docId, _forwardIndexMap.get(column), _dictionaryMap.get(column),
+ _numValuesInfoMap.get(column).getMaxNumValuesPerMVEntry());
reuse.putValue(column, value);
if (_nullHandlingEnabled) {
@@ -745,6 +727,45 @@ public class MutableSegmentImpl implements MutableSegment {
return reuse;
}
+ /**
+ * Helper method to read the value for the given document id.
+ */
+ private static Object getValue(int docId, MutableForwardIndex forwardIndex,
+ @Nullable BaseMutableDictionary dictionary, int maxNumMultiValues) {
+ if (dictionary != null) {
+ // Dictionary based
+ if (forwardIndex.isSingleValue()) {
+ int dictId = forwardIndex.getDictId(docId);
+ return dictionary.get(dictId);
+ } else {
+ int[] dictIds = new int[maxNumMultiValues];
+ int numValues = forwardIndex.getDictIdMV(docId, dictIds);
+ Object[] value = new Object[numValues];
+ for (int i = 0; i < numValues; i++) {
+ value[i] = dictionary.get(dictIds[i]);
+ }
+ return value;
+ }
+ } else {
+ // Raw index based
+ // TODO: support multi-valued column
+ switch (forwardIndex.getValueType()) {
+ case INT:
+ return forwardIndex.getInt(docId);
+ case LONG:
+ return forwardIndex.getLong(docId);
+ case FLOAT:
+ return forwardIndex.getFloat(docId);
+ case DOUBLE:
+ return forwardIndex.getDouble(docId);
+ case STRING:
+ return forwardIndex.getString(docId);
+ default:
+ throw new IllegalStateException();
+ }
+ }
+ }
+
@Override
public void destroy() {
_logger.info("Trying to close RealtimeSegmentImpl : {}", _segmentName);
@@ -775,16 +796,16 @@ public class MutableSegmentImpl implements MutableSegment {
}
}
- for (DataFileReader dfReader : _indexReaderWriterMap.values()) {
+ for (MutableForwardIndex mutableForwardIndex : _forwardIndexMap.values()) {
try {
- dfReader.close();
+ mutableForwardIndex.close();
} catch (IOException e) {
_logger.error("Failed to close index. Service will continue with potential memory leak, error: ", e);
// fall through to close other segments
}
}
// clear map now that index is closed to prevent accidental usage
- _indexReaderWriterMap.clear();
+ _forwardIndexMap.clear();
for (InvertedIndexReader index : _invertedIndexMap.values()) {
if (index instanceof RealtimeInvertedIndexReader) {
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/BaseSingleColumnMultiValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/BaseSingleColumnMultiValueReader.java
deleted file mode 100644
index 2b46b12..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/BaseSingleColumnMultiValueReader.java
+++ /dev/null
@@ -1,64 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader;
-
-public abstract class BaseSingleColumnMultiValueReader<T extends ReaderContext> implements SingleColumnMultiValueReader<T> {
-
- public int getCharArray(int row, char[] charArray) {
- throw new UnsupportedOperationException();
- }
-
- public int getShortArray(int row, short[] shortsArray) {
- throw new UnsupportedOperationException();
- }
-
- public int getIntArray(int row, int[] intArray) {
- throw new UnsupportedOperationException();
- }
-
- public int getLongArray(int row, long[] longArray) {
- throw new UnsupportedOperationException();
- }
-
- public int getFloatArray(int row, float[] floatArray) {
- throw new UnsupportedOperationException();
- }
-
- public int getDoubleArray(int row, double[] doubleArray) {
- throw new UnsupportedOperationException();
- }
-
- public int getStringArray(int row, String[] stringArray) {
- throw new UnsupportedOperationException();
- }
-
- public int getBytesArray(int row, byte[][] bytesArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public T createContext() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getIntArray(int row, int[] intArray, T readerContext) {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/BaseSingleColumnSingleValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/BaseSingleColumnSingleValueReader.java
deleted file mode 100644
index ff50edf..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/BaseSingleColumnSingleValueReader.java
+++ /dev/null
@@ -1,102 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader;
-
-import java.io.IOException;
-
-
-public abstract class BaseSingleColumnSingleValueReader<T extends ReaderContext> implements SingleColumnSingleValueReader<T> {
-
- @Override
- public void close()
- throws IOException {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public char getChar(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public short getShort(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getInt(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getInt(int rowId, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long getLong(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long getLong(int rowId, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public float getFloat(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public float getFloat(int rowId, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public double getDouble(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public double getDouble(int rowId, T context) {
- throw new UnsupportedOperationException();
- }
-
- public String getString(int row) {
- throw new UnsupportedOperationException();
- }
-
- public String getString(int row, T context) {
- throw new UnsupportedOperationException();
- }
-
- public byte[] getBytes(int row) {
- throw new UnsupportedOperationException();
- }
-
- public byte[] getBytes(int row, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void readValues(int[] rows, int rowStartPos, int rowSize, int[] values, int valuesStartPos) {
- throw new UnsupportedOperationException("not supported");
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/DataFileReader.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/DataFileReader.java
deleted file mode 100644
index 4a24788..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/DataFileReader.java
+++ /dev/null
@@ -1,30 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader;
-
-import java.io.Closeable;
-
-
-/**
- * Marker interface to read any file in the segment directory
- */
-public interface DataFileReader<T extends ReaderContext> extends Closeable {
-
- T createContext();
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/ReaderContext.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/ReaderContext.java
deleted file mode 100644
index 3d5c174..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/ReaderContext.java
+++ /dev/null
@@ -1,23 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader;
-
-public interface ReaderContext {
-
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleColumnMultiValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleColumnMultiValueReader.java
deleted file mode 100644
index f7fa5a7..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleColumnMultiValueReader.java
+++ /dev/null
@@ -1,81 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader;
-
-public interface SingleColumnMultiValueReader<T extends ReaderContext> extends DataFileReader<T> {
-
- /**
- * Read the multiple values for a column at a specific row.
- * @param row
- * @param charArray
- * @return returns the number of chars read
- */
- int getCharArray(int row, char[] charArray);
-
- /**
- * @param row
- * @param shortsArray
- * @return return the number of shorts read
- */
- int getShortArray(int row, short[] shortsArray);
-
- /**
- * @param row
- * @param intArray
- * @return
- */
- int getIntArray(int row, int[] intArray);
-
- /**
- * @param row
- * @param longArray
- * @return
- */
- int getLongArray(int row, long[] longArray);
-
- /**
- * @param row
- * @param floatArray
- * @return
- */
- int getFloatArray(int row, float[] floatArray);
-
- /**
- * @param row
- * @param doubleArray
- * @return
- */
- int getDoubleArray(int row, double[] doubleArray);
-
- /**
- * @param row
- * @param stringArray
- * @return
- */
- int getStringArray(int row, String[] stringArray);
-
- /**
- * @param row
- * @param bytesArray
- * @return
- */
- int getBytesArray(int row, byte[][] bytesArray);
-
- int getIntArray(int row, int[] intArray, T readerContext);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleColumnSingleValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleColumnSingleValueReader.java
deleted file mode 100644
index 680a5ed..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleColumnSingleValueReader.java
+++ /dev/null
@@ -1,151 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader;
-
-public interface SingleColumnSingleValueReader<T extends ReaderContext> extends DataFileReader<T> {
-
- /**
- * Fetch the char at a row.
- *
- * @param row Row for which to fetch the value.
- * @return Char value at row.
- */
- char getChar(int row);
-
- /**
- * Fetch short value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @return Short value at row.
- */
- short getShort(int row);
-
- /**
- * Fetch the int value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @return int value at row.
- */
- int getInt(int row);
-
- /**
- * Fetch the int value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @param context Reader context.
- * @return int value at row.
- */
- int getInt(int row, T context);
-
- /**
- * Fetch the long value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @return value value at row.
- */
- long getLong(int row);
-
- /**
- * Fetch the long value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @param context Reader context.
- * @return long value at row.
- */
- long getLong(int row, T context);
-
- /**
- * Fetch the float value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @return float value at row.
- */
- float getFloat(int row);
-
- /**
- * Fetch the float value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @param context Reader context.
- * @return float value at row.
- */
- float getFloat(int row, T context);
-
- /**
- * Fetch the double value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @return double value at row.
- */
- double getDouble(int row);
-
- /**
- * Fetch the double value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @param context Reader context.
- * @return double value at row.
- */
- double getDouble(int row, T context);
-
- /**
- * Fetch the String value at the specified row.
- *
- * @param row Row for which to fetch the value.
- * @return String value at row.
- */
- String getString(int row);
-
- /**
- * Fetch String value for the given row.
- *
- * @param row Row for which to get the string.
- * @param context Reader context.
- * @return String at row.
- */
- String getString(int row, T context);
-
- /**
- * Fetch the byte[] value for the given row.
- *
- * @param row Row for which to get the byte[].
- * @return byte[] value at row.
- */
- byte[] getBytes(int row);
-
- /**
- * Fetch the byte[] value for the given row.
- *
- * @param row Row for which to get the byte[].
- * @param context Reader context.
- * @return byte[] byte[] at the given row
- */
- byte[] getBytes(int row, T context);
-
- /**
- * Read a specified number of values from a given start offset.
- *
- * @param rows Array containing rows id's to read.
- * @param rowStartPos Start offset in 'rows' array.
- * @param rowSize Number of elements to read from 'rows'.
- * @param values Output array
- * @param valuesStartPos Start offset of 'values' array to write the values.
- */
- void readValues(int[] rows, int rowStartPos, int rowSize, int[] values, int valuesStartPos);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleValueMultiColumnReader.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleValueMultiColumnReader.java
deleted file mode 100644
index 1d21773..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/SingleValueMultiColumnReader.java
+++ /dev/null
@@ -1,156 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader;
-
-/**
- * Interface for reader for Single Value and Multiple Columns.
- *
- * @param <T>
- */
-public interface SingleValueMultiColumnReader<T extends ReaderContext> extends DataFileReader<T> {
-
- /**
- * Get the char at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @return Char value at (row, column)
- */
- char getChar(int row, int column);
-
- /**
- * Get the short at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @return Short value at (row, column)
- */
- short getShort(int row, int column);
-
- /**
- * Get the int at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @return int value at (row, column)
- */
- int getInt(int row, int column);
-
- /**
- * Get the int at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @param context Reader context
- * @return int value at (row, column)
- */
- int getInt(int row, int column, T context);
-
- /**
- * Get the long at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @return long value at (row, column)
- */
- long getLong(int row, int column);
-
- /**
- * Get the long at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @param context Reader context
- *
- * @return long value at (row, column)
- */
- long getLong(int row, int column, T context);
-
- /**
- * Get the float at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @return float value at (row, column)
- */
- float getFloat(int row, int column);
-
- /**
- * Get the float at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @param context Reader context
- *
- * @return float value at (row, column)
- */
- float getFloat(int row, int column, T context);
-
- /**
- * Get the double at a given (row, column).
- *
- * @param row row id
- * @param column column id
- *
- * @return double value at (row, column)
- */
- double getDouble(int row, int column);
-
- /**
- * Get the double at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @param context Reader context
- *
- * @return double value at (row, column)
- */
- double getDouble(int row, int column, T context);
-
- /**
- * Get the String at a given (row, column).
- *
- * @param row row id
- * @param column column id
- * @return String value at (row, column)
- */
- String getString(int row, int column);
-
- /**
- * Get the String at a given (row, column).
- *
- * @param row row id
- * @param column column id
- *
- * @param context Reader context
- *
- * @return String value at (row, column)
- */
- String getString(int row, int column, T context);
-
- /**
- * Get the byte at a given (row, column).
- *
- * @param row row id
- * @param column column id
- *
- * @return byte value at (row, column)
- */
- byte[] getBytes(int row, int column);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ChunkReaderContext.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ChunkReaderContext.java
deleted file mode 100644
index 5fb1cf7..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ChunkReaderContext.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader.impl;
-
-import java.nio.ByteBuffer;
-
-
-/**
- * Class to represent the reader context for ChunkReaders.
- * Information saved in the context can be used by subsequent reads as cache.
- * <ul>
- * <li> Chunk Buffer from the previous read. Useful if the subsequent read is from the same buffer,
- * as it avoids chunk decompression. </li>
- * <li> Id for the chunk </li>
- * </ul>
- */
-public class ChunkReaderContext extends UnSortedValueReaderContext {
- int _chunkId;
- ByteBuffer _chunkBuffer;
-
- public ChunkReaderContext(int maxChunkSize) {
- _chunkBuffer = ByteBuffer.allocateDirect(maxChunkSize);
- _chunkId = -1;
- }
-
- public ByteBuffer getChunkBuffer() {
- return _chunkBuffer;
- }
-
- public int getChunkId() {
- return _chunkId;
- }
-
- public void setChunkId(int chunkId) {
- _chunkId = chunkId;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantSVSortedIndex.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantSVSortedIndex.java
deleted file mode 100644
index 98c68fa..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantSVSortedIndex.java
+++ /dev/null
@@ -1,73 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader.impl;
-
-import java.io.IOException;
-import java.util.Arrays;
-import org.apache.pinot.common.utils.Pairs;
-import org.apache.pinot.core.io.reader.BaseSingleColumnSingleValueReader;
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.reader.impl.v1.SortedIndexReader;
-
-
-/**
- * Sorted index for single-value column with constant values.
- */
-public class ConstantSVSortedIndex extends BaseSingleColumnSingleValueReader<ReaderContext> implements SortedIndexReader<ReaderContext> {
- private final int _numDocs;
-
- public ConstantSVSortedIndex(int numDocs) {
- _numDocs = numDocs;
- }
-
- @Override
- public int getInt(int row) {
- return 0;
- }
-
- @Override
- public int getInt(int rowId, ReaderContext context) {
- return 0;
- }
-
- @Override
- public void readValues(int[] rows, int rowStartPos, int rowSize, int[] values, int valuesStartPos) {
- Arrays.fill(values, valuesStartPos, valuesStartPos + rowSize, 0);
- }
-
- @Override
- public ReaderContext createContext() {
- return null;
- }
-
- @Override
- public Pairs.IntPair getDocIds(int dictId) {
- return new Pairs.IntPair(0, _numDocs - 1);
- }
-
- @Override
- public Pairs.IntPair getDocIds(Object value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void close()
- throws IOException {
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/UnSortedValueReaderContext.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/UnSortedValueReaderContext.java
deleted file mode 100644
index 4655885..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/UnSortedValueReaderContext.java
+++ /dev/null
@@ -1,26 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader.impl;
-
-import org.apache.pinot.core.io.reader.ReaderContext;
-
-
-public class UnSortedValueReaderContext implements ReaderContext {
-
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedBitSingleValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedBitSingleValueReader.java
deleted file mode 100644
index 5512647..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedBitSingleValueReader.java
+++ /dev/null
@@ -1,61 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader.impl.v1;
-
-import org.apache.pinot.core.io.reader.BaseSingleColumnSingleValueReader;
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.util.FixedBitIntReaderWriter;
-import org.apache.pinot.core.segment.memory.PinotDataBuffer;
-
-
-public final class FixedBitSingleValueReader extends BaseSingleColumnSingleValueReader {
- private final FixedBitIntReaderWriter _reader;
-
- public FixedBitSingleValueReader(PinotDataBuffer dataBuffer, int numRows, int numBitsPerValue) {
- _reader = new FixedBitIntReaderWriter(dataBuffer, numRows, numBitsPerValue);
- }
-
- @Override
- public int getInt(int row) {
- return _reader.readInt(row);
- }
-
- @Override
- public int getInt(int row, ReaderContext context) {
- return _reader.readInt(row);
- }
-
- @Override
- public void readValues(int[] rows, int rowsStartIndex, int rowSize, int[] values, int valuesStartIndex) {
- int rowsEndIndex = rowsStartIndex + rowSize;
- for (int i = rowsStartIndex; i < rowsEndIndex; i++) {
- values[valuesStartIndex++] = getInt(rows[i]);
- }
- }
-
- @Override
- public ReaderContext createContext() {
- return null;
- }
-
- @Override
- public void close() {
- _reader.close();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedByteChunkSingleValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedByteChunkSingleValueReader.java
deleted file mode 100644
index b367263..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedByteChunkSingleValueReader.java
+++ /dev/null
@@ -1,142 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.reader.impl.v1;
-
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import org.apache.pinot.core.io.reader.impl.ChunkReaderContext;
-import org.apache.pinot.core.io.writer.impl.v1.FixedByteChunkSingleValueWriter;
-import org.apache.pinot.core.segment.memory.PinotDataBuffer;
-
-
-/**
- * Reader class for data written out by {@link FixedByteChunkSingleValueWriter}.
- * For data layout, please refer to the documentation for {@link FixedByteChunkSingleValueReader}
- *
- */
-public class FixedByteChunkSingleValueReader extends BaseChunkSingleValueReader {
-
- // Thread local (reusable) byte[] to read bytes from data file.
- private final ThreadLocal<byte[]> _reusableBytes = ThreadLocal.withInitial(() -> new byte[_lengthOfLongestEntry]);
-
- /**
- * Constructor for the class.
- *
- * @param pinotDataBuffer Data buffer to read from
- * @throws IOException
- */
- public FixedByteChunkSingleValueReader(PinotDataBuffer pinotDataBuffer) {
- super(pinotDataBuffer);
- }
-
- @Override
- public int getInt(int row) {
- if (!isCompressed()) {
- return getRawData().getInt(row * Integer.BYTES);
- } else {
- throw new UnsupportedOperationException("Read without context not supported for compressed data.");
- }
- }
-
- @Override
- public int getInt(int row, ChunkReaderContext context) {
- assert _lengthOfLongestEntry == Integer.BYTES;
- int chunkRowId = row % _numDocsPerChunk;
- ByteBuffer chunkBuffer = getChunkForRow(row, context);
- return chunkBuffer.getInt(chunkRowId * Integer.BYTES);
- }
-
- @Override
- public float getFloat(int row) {
- if (!isCompressed()) {
- return getRawData().getFloat(row * Float.BYTES);
- } else {
- throw new UnsupportedOperationException("Read without context not supported for compressed data.");
- }
- }
-
- @Override
- public float getFloat(int row, ChunkReaderContext context) {
- assert _lengthOfLongestEntry == Float.BYTES;
- int chunkRowId = row % _numDocsPerChunk;
- ByteBuffer chunkBuffer = getChunkForRow(row, context);
- return chunkBuffer.getFloat(chunkRowId * Float.BYTES);
- }
-
- @Override
- public long getLong(int row) {
- if (!isCompressed()) {
- return getRawData().getLong(row * Long.BYTES);
- } else {
- throw new UnsupportedOperationException("Read without context not supported for compressed data.");
- }
- }
-
- @Override
- public long getLong(int row, ChunkReaderContext context) {
- assert _lengthOfLongestEntry == Long.BYTES;
- int chunkRowId = row % _numDocsPerChunk;
- ByteBuffer chunkBuffer = getChunkForRow(row, context);
- return chunkBuffer.getLong(chunkRowId * Long.BYTES);
- }
-
- @Override
- public double getDouble(int row) {
- if (!isCompressed()) {
- return getRawData().getDouble(row * Double.BYTES);
- } else {
- throw new UnsupportedOperationException("Read without context not supported for compressed data.");
- }
- }
-
- @Override
- public double getDouble(int row, ChunkReaderContext context) {
- assert _lengthOfLongestEntry == Double.BYTES;
- int chunkRowId = row % _numDocsPerChunk;
- ByteBuffer chunkBuffer = getChunkForRow(row, context);
- return chunkBuffer.getDouble(chunkRowId * Double.BYTES);
- }
-
- @Override
- public byte[] getBytes(int row) {
- if (!isCompressed()) {
- byte[] bytes = _reusableBytes.get();
- getRawData().copyTo(row * _lengthOfLongestEntry, bytes, 0, _lengthOfLongestEntry);
- return bytes;
- } else {
- throw new UnsupportedOperationException("Read without context not supported for compressed data.");
- }
- }
-
- @Override
- public byte[] getBytes(int row, ChunkReaderContext context) {
- int chunkRowId = row % _numDocsPerChunk;
- ByteBuffer chunkBuffer = getChunkForRow(row, context);
-
- byte[] bytes = _reusableBytes.get();
- chunkBuffer.position(chunkRowId * _lengthOfLongestEntry);
- chunkBuffer.get(bytes);
- return bytes;
- }
-
- @Override
- public ChunkReaderContext createContext() {
- return new ChunkReaderContext(_chunkSize);
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleColumnMultiValueReaderWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleColumnMultiValueReaderWriter.java
deleted file mode 100644
index 81a1869..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleColumnMultiValueReaderWriter.java
+++ /dev/null
@@ -1,117 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.readerwriter;
-
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.reader.SingleColumnMultiValueReader;
-import org.apache.pinot.core.io.writer.SingleColumnMultiValueWriter;
-
-
-public abstract class BaseSingleColumnMultiValueReaderWriter<T extends ReaderContext> implements SingleColumnMultiValueReader<T>, SingleColumnMultiValueWriter {
-
- @Override
- public int getCharArray(int row, char[] charArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getShortArray(int row, short[] shortsArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getIntArray(int row, int[] intArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getLongArray(int row, long[] longArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getFloatArray(int row, float[] floatArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getDoubleArray(int row, double[] doubleArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getStringArray(int row, String[] stringArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getBytesArray(int row, byte[][] bytesArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public T createContext() {
- return null;
- }
-
- @Override
- public int getIntArray(int row, int[] intArray, T readerContext) {
- return getIntArray(row, intArray);
- }
-
- @Override
- public void setCharArray(int row, char[] charArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setShortArray(int row, short[] shortsArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setIntArray(int row, int[] intArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setLongArray(int row, long[] longArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setFloatArray(int row, float[] floatArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setDoubleArray(int row, double[] doubleArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setStringArray(int row, String[] stringArray) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setBytesArray(int row, byte[][] bytesArray) {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleColumnSingleValueReaderWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleColumnSingleValueReaderWriter.java
deleted file mode 100644
index 7dc8758..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleColumnSingleValueReaderWriter.java
+++ /dev/null
@@ -1,155 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.readerwriter;
-
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.core.io.writer.SingleColumnSingleValueWriter;
-
-
-public abstract class BaseSingleColumnSingleValueReaderWriter<T extends ReaderContext> implements SingleColumnSingleValueReader<T>, SingleColumnSingleValueWriter {
-
- @Override
- public char getChar(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public short getShort(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getInt(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getInt(int row, ReaderContext context) {
- return getInt(row);
- }
-
- @Override
- public long getLong(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long getLong(int row, ReaderContext context) {
- return getLong(row);
- }
-
- @Override
- public float getFloat(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public float getFloat(int row, ReaderContext context) {
- return getFloat(row);
- }
-
- @Override
- public double getDouble(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public double getDouble(int row, ReaderContext context) {
- return getDouble(row);
- }
-
- @Override
- public String getString(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public String getString(int row, ReaderContext context) {
- return getString(row);
- }
-
- @Override
- public byte[] getBytes(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public byte[] getBytes(int row, ReaderContext context) {
- return getBytes(row);
- }
-
- @Override
- public void readValues(int[] rows, int rowStartPos, int rowSize, int[] values, int valuesStartPos) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public T createContext() {
- return null;
- }
-
- @Override
- public void setChar(int row, char ch) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setInt(int row, int i) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setShort(int row, short s) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setLong(int row, long l) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setFloat(int row, float f) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setDouble(int row, double d) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setString(int row, String string) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setBytes(int row, byte[] bytes) {
- throw new UnsupportedOperationException();
- }
-
- public int getLengthOfShortestElement() {
- throw new UnsupportedOperationException();
- }
-
- public int getLengthOfLongestElement() {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleValueMultiColumnReaderWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleValueMultiColumnReaderWriter.java
deleted file mode 100644
index 60d4a32..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/BaseSingleValueMultiColumnReaderWriter.java
+++ /dev/null
@@ -1,132 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.readerwriter;
-
-import java.io.IOException;
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.reader.SingleValueMultiColumnReader;
-import org.apache.pinot.core.io.writer.SingleValueMultiColumnWriter;
-
-
-/**
- * Abstract class for reader and writer interfaces for Single-value Multiple-Columns case.
- * @param <T>
- */
-public abstract class BaseSingleValueMultiColumnReaderWriter<T extends ReaderContext> implements SingleValueMultiColumnReader<T>, SingleValueMultiColumnWriter {
- @Override
- public char getChar(int row, int column) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public short getShort(int row, int column) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getInt(int row, int column) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getInt(int row, int column, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long getLong(int row, int column) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public long getLong(int row, int column, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public float getFloat(int row, int column) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public float getFloat(int row, int column, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public double getDouble(int row, int column) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public double getDouble(int row, int column, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public String getString(int row, int column) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public String getString(int row, int column, T context) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public byte[] getBytes(int row, int column) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public T createContext() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setInt(int row, int column, int value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setLong(int row, int column, long value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setFloat(int row, int column, float value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setDouble(int row, int column, double value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setString(int row, int column, String value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void close()
- throws IOException {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleValueMultiColumnReaderWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleValueMultiColumnReaderWriter.java
index c125856..63ab271 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleValueMultiColumnReaderWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleValueMultiColumnReaderWriter.java
@@ -18,23 +18,24 @@
*/
package org.apache.pinot.core.io.readerwriter.impl;
+import java.io.Closeable;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.pinot.core.io.reader.impl.FixedByteSingleValueMultiColReader;
-import org.apache.pinot.core.io.readerwriter.BaseSingleValueMultiColumnReaderWriter;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
import org.apache.pinot.core.io.writer.impl.FixedByteSingleValueMultiColWriter;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
/**
- * Extension of {@link BaseSingleValueMultiColumnReaderWriter} that fully implements a
* SingleValue MultiColumn reader and writer for fixed size columns.
*
* This implementation currently does not support reading with context, but can be enhanced to do so.
+ *
+ * TODO: Clean this up
*/
-public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValueMultiColumnReaderWriter {
+public class FixedByteSingleValueMultiColumnReaderWriter implements Closeable {
private final List<FixedByteSingleValueMultiColWriter> _writers;
private volatile List<FixedByteSingleValueMultiColReader> _readers;
@@ -76,7 +77,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
_chunkSizeInBytes = rowSizeInBytes * numRowsPerChunk;
}
- @Override
public int getInt(int row, int column) {
int rowInChunk = row % _numRowsPerChunk;
int chunkId = row / _numRowsPerChunk;
@@ -84,7 +84,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
return _readers.get(chunkId).getInt(rowInChunk, column);
}
- @Override
public long getLong(int row, int column) {
int rowInChunk = row % _numRowsPerChunk;
int chunkId = row / _numRowsPerChunk;
@@ -92,7 +91,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
return _readers.get(chunkId).getLong(rowInChunk, column);
}
- @Override
public float getFloat(int row, int column) {
int rowInChunk = row % _numRowsPerChunk;
int chunkId = row / _numRowsPerChunk;
@@ -100,7 +98,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
return _readers.get(chunkId).getFloat(rowInChunk, column);
}
- @Override
public double getDouble(int row, int column) {
int rowInChunk = row % _numRowsPerChunk;
int chunkId = row / _numRowsPerChunk;
@@ -108,7 +105,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
return _readers.get(chunkId).getDouble(rowInChunk, column);
}
- @Override
public String getString(int row, int column) {
int rowInChunk = row % _numRowsPerChunk;
int chunkId = row / _numRowsPerChunk;
@@ -116,7 +112,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
return _readers.get(chunkId).getString(rowInChunk, column);
}
- @Override
public void setInt(int row, int column, int value) {
ensureCapacity(row);
int rowInChunk = row % _numRowsPerChunk;
@@ -124,7 +119,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
_writers.get(chunkId).setInt(rowInChunk, column, value);
}
- @Override
public void setLong(int row, int column, long value) {
ensureCapacity(row);
int rowInChunk = row % _numRowsPerChunk;
@@ -132,7 +126,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
_writers.get(chunkId).setLong(rowInChunk, column, value);
}
- @Override
public void setFloat(int row, int column, float value) {
ensureCapacity(row);
int rowInChunk = row % _numRowsPerChunk;
@@ -140,7 +133,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
_writers.get(chunkId).setFloat(rowInChunk, column, value);
}
- @Override
public void setDouble(int row, int column, double value) {
ensureCapacity(row);
int rowInChunk = row % _numRowsPerChunk;
@@ -148,7 +140,6 @@ public class FixedByteSingleValueMultiColumnReaderWriter extends BaseSingleValue
_writers.get(chunkId).setDouble(rowInChunk, column, value);
}
- @Override
public void setString(int row, int column, String value) {
ensureCapacity(row);
int rowInChunk = row % _numRowsPerChunk;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleColumnMultiValueWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleColumnMultiValueWriter.java
deleted file mode 100644
index 7edac92..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleColumnMultiValueWriter.java
+++ /dev/null
@@ -1,78 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.writer;
-
-public interface SingleColumnMultiValueWriter extends DataFileWriter {
- /**
- * Read the multiple values for a column at a specific row.
- *
- * @param row
- * @param charArray
- */
- void setCharArray(int row, char[] charArray);
-
- /**
- *
- * @param row
- * @param shortsArray
- */
- void setShortArray(int row, short[] shortsArray);
-
- /**
- *
- * @param row
- * @param intArray
- */
- void setIntArray(int row, int[] intArray);
-
- /**
- *
- * @param row
- * @param longArray
- */
- void setLongArray(int row, long[] longArray);
-
- /**
- *
- * @param row
- * @param floatArray
- */
- void setFloatArray(int row, float[] floatArray);
-
- /**
- *
- * @param row
- * @param doubleArray
- */
- void setDoubleArray(int row, double[] doubleArray);
-
- /**
- *
- * @param row
- * @param stringArray
- */
- void setStringArray(int row, String[] stringArray);
-
- /**
- *
- * @param row
- * @param bytesArray
- */
- void setBytesArray(int row, byte[][] bytesArray);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleColumnSingleValueWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleColumnSingleValueWriter.java
deleted file mode 100644
index e5d13bc..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleColumnSingleValueWriter.java
+++ /dev/null
@@ -1,77 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.writer;
-
-public interface SingleColumnSingleValueWriter extends DataFileWriter {
- /**
- *
- * @param row
- * @param ch
- */
- public void setChar(int row, char ch);
-
- /**
- *
- * @param row
- * @param i
- */
- public void setInt(int row, int i);
-
- /**
- *
- * @param row
- * @param s
- */
- public void setShort(int row, short s);
-
- /**
- *
- * @param row
- * @param l
- */
- public void setLong(int row, long l);
-
- /**
- *
- * @param row
- * @param f
- */
- public void setFloat(int row, float f);
-
- /**
- *
- * @param row
- * @param d
- */
- public void setDouble(int row, double d);
-
- /**
- *
- * @param row
- * @param string
- */
- public void setString(int row, String string);
-
- /**
- *
- * @param row
- * @param bytes
- */
- public void setBytes(int row, byte[] bytes);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleValueMultiColumnWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleValueMultiColumnWriter.java
deleted file mode 100644
index 2e072a7..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/SingleValueMultiColumnWriter.java
+++ /dev/null
@@ -1,70 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.writer;
-
-/**
- * Interface for writing multiple columns with single values.
- */
-public interface SingleValueMultiColumnWriter extends DataFileWriter {
-
- /**
- * Set the given int value at the specified (row, column).
- *
- * @param row Row id.
- * @param column Column id.
- * @param value Value to be set.
- */
- void setInt(int row, int column, int value);
-
- /**
- * Set the given long value at the specified (row, column).
- *
- * @param row Row id.
- * @param column Column id.
- * @param value Value to be set.
- */
- void setLong(int row, int column, long value);
-
- /**
- * Set the given float value at the specified (row, column).
- *
- * @param row Row id.
- * @param column Column id.
- * @param value Value to be set.
- */
- void setFloat(int row, int column, float value);
-
- /**
- * Set the given double value at the specified (row, column).
- *
- * @param row Row id.
- * @param column Column id.
- * @param value Value to be set.
- */
- void setDouble(int row, int column, double value);
-
- /**
- * Set the given String value at the specified (row, column).
- *
- * @param row Row id.
- * @param column Column id.
- * @param value Value to be set.
- */
- void setString(int row, int column, String value);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/BaseChunkSingleValueWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/BaseChunkSVForwardIndexWriter.java
similarity index 80%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/BaseChunkSingleValueWriter.java
rename to pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/BaseChunkSVForwardIndexWriter.java
index c51a465..316a984 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/BaseChunkSingleValueWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/BaseChunkSVForwardIndexWriter.java
@@ -16,9 +16,10 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.writer.impl.v1;
+package org.apache.pinot.core.io.writer.impl;
import com.google.common.base.Preconditions;
+import java.io.Closeable;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
@@ -27,21 +28,19 @@ import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import org.apache.pinot.core.io.compression.ChunkCompressor;
import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
-import org.apache.pinot.core.io.writer.SingleColumnSingleValueWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Abstract implementation for {@link SingleColumnSingleValueWriter}
- * Base class for fixed and variable byte writer implementations.
+ * Base implementation for chunk-based single-value raw (non-dictionary-encoded) forward index writer.
*/
-public abstract class BaseChunkSingleValueWriter implements SingleColumnSingleValueWriter {
+public abstract class BaseChunkSVForwardIndexWriter implements Closeable {
// TODO: Remove this before release 0.5.0
public static final int DEFAULT_VERSION = 2;
public static final int CURRENT_VERSION = 3;
- private static final Logger LOGGER = LoggerFactory.getLogger(BaseChunkSingleValueWriter.class);
+ private static final Logger LOGGER = LoggerFactory.getLogger(BaseChunkSVForwardIndexWriter.class);
private static final int FILE_HEADER_ENTRY_CHUNK_OFFSET_SIZE_V1V2 = Integer.BYTES;
private static final int FILE_HEADER_ENTRY_CHUNK_OFFSET_SIZE_V3 = Long.BYTES;
@@ -68,8 +67,8 @@ public abstract class BaseChunkSingleValueWriter implements SingleColumnSingleVa
* @param version version of File
* @throws FileNotFoundException
*/
- protected BaseChunkSingleValueWriter(File file, ChunkCompressorFactory.CompressionType compressionType, int totalDocs,
- int numDocsPerChunk, int chunkSize, int sizeOfEntry, int version)
+ protected BaseChunkSVForwardIndexWriter(File file, ChunkCompressorFactory.CompressionType compressionType,
+ int totalDocs, int numDocsPerChunk, int chunkSize, int sizeOfEntry, int version)
throws FileNotFoundException {
Preconditions.checkArgument(version == DEFAULT_VERSION || version == CURRENT_VERSION);
_chunkSize = chunkSize;
@@ -94,46 +93,6 @@ public abstract class BaseChunkSingleValueWriter implements SingleColumnSingleVa
}
@Override
- public void setChar(int row, char ch) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setInt(int row, int i) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setShort(int row, short s) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setLong(int row, long l) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setFloat(int row, float f) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setDouble(int row, double d) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setString(int row, String string) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setBytes(int row, byte[] bytes) {
- throw new UnsupportedOperationException();
- }
-
- @Override
public void close()
throws IOException {
@@ -219,7 +178,7 @@ public abstract class BaseChunkSingleValueWriter implements SingleColumnSingleVa
}
if (_headerEntryChunkOffsetSize == Integer.BYTES) {
- _header.putInt((int)_dataOffset);
+ _header.putInt((int) _dataOffset);
} else if (_headerEntryChunkOffsetSize == Long.BYTES) {
_header.putLong(_dataOffset);
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedBitMultiValueWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedBitMVForwardIndexWriter.java
similarity index 69%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedBitMultiValueWriter.java
rename to pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedBitMVForwardIndexWriter.java
index f681779..a9d9a29 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedBitMultiValueWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedBitMVForwardIndexWriter.java
@@ -16,22 +16,23 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.writer.impl.v1;
+package org.apache.pinot.core.io.writer.impl;
import com.google.common.base.Preconditions;
+import java.io.Closeable;
import java.io.File;
import java.io.IOException;
import java.nio.ByteOrder;
import org.apache.pinot.core.io.util.FixedBitIntReaderWriter;
import org.apache.pinot.core.io.util.FixedByteValueReaderWriter;
import org.apache.pinot.core.io.util.PinotDataBitSet;
-import org.apache.pinot.core.io.writer.SingleColumnMultiValueWriter;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
/**
+ * Bit-compressed dictionary-encoded forward index writer for multi-value columns. The values written are dictionary
+ * ids.
+ *
* Storage Layout
* ==============
* There will be three sections HEADER section, BITMAP and RAW DATA
@@ -49,11 +50,9 @@ import org.slf4j.LoggerFactory;
* Over all each look up will take log(NUM CHUNKS) for binary search + CHUNK to linear scan on the
* bitmap to find the right offset in the raw data section
*/
-public class FixedBitMultiValueWriter implements SingleColumnMultiValueWriter {
- private static final Logger LOGGER = LoggerFactory.getLogger(FixedBitMultiValueWriter.class);
-
- private static int SIZE_OF_INT = 4;
- private static int NUM_COLS_IN_HEADER = 1;
+public class FixedBitMVForwardIndexWriter implements Closeable {
+ private static final int SIZE_OF_INT = 4;
+ private static final int NUM_COLS_IN_HEADER = 1;
private static final int PREFERRED_NUM_VALUES_PER_CHUNK = 2048;
private PinotDataBuffer indexDataBuffer;
@@ -67,21 +66,22 @@ public class FixedBitMultiValueWriter implements SingleColumnMultiValueWriter {
private int numChunks;
int prevRowStartIndex = 0;
int prevRowLength = 0;
- int prevRowId = -1;
private int chunkOffsetHeaderSize;
private int bitsetSize;
private long rawDataSize;
private long totalSize;
private int docsPerChunk;
- public FixedBitMultiValueWriter(File file, int numDocs, int totalNumValues, int columnSizeInBits)
+ private int _nextDocId = 0;
+
+ public FixedBitMVForwardIndexWriter(File file, int numDocs, int totalNumValues, int numBitsPerValue)
throws Exception {
float averageValuesPerDoc = totalNumValues / numDocs;
this.docsPerChunk = (int) (Math.ceil(PREFERRED_NUM_VALUES_PER_CHUNK / averageValuesPerDoc));
this.numChunks = (numDocs + docsPerChunk - 1) / docsPerChunk;
chunkOffsetHeaderSize = numChunks * SIZE_OF_INT * NUM_COLS_IN_HEADER;
bitsetSize = (totalNumValues + 7) / 8;
- rawDataSize = ((long) totalNumValues * columnSizeInBits + 7) / 8;
+ rawDataSize = ((long) totalNumValues * numBitsPerValue + 7) / 8;
totalSize = chunkOffsetHeaderSize + bitsetSize + rawDataSize;
Preconditions.checkState(totalSize > 0 && totalSize < Integer.MAX_VALUE, "Total size can not exceed 2GB for file: ",
file.toString());
@@ -96,7 +96,7 @@ public class FixedBitMultiValueWriter implements SingleColumnMultiValueWriter {
chunkOffsetsWriter = new FixedByteValueReaderWriter(chunkOffsetsBuffer);
customBitSet = new PinotDataBitSet(bitsetBuffer);
- rawDataWriter = new FixedBitIntReaderWriter(rawDataBuffer, totalNumValues, columnSizeInBits);
+ rawDataWriter = new FixedBitIntReaderWriter(rawDataBuffer, totalNumValues, numBitsPerValue);
}
public int getChunkOffsetHeaderSize() {
@@ -139,57 +139,20 @@ public class FixedBitMultiValueWriter implements SingleColumnMultiValueWriter {
rawDataWriter = null;
}
- private int updateHeader(int rowId, int length) {
- assert (rowId == prevRowId + 1);
+ private int updateHeader(int length) {
int newStartIndex = prevRowStartIndex + prevRowLength;
- if (rowId % docsPerChunk == 0) {
- int chunkId = rowId / docsPerChunk;
+ int docId = _nextDocId++;
+ if (docId % docsPerChunk == 0) {
+ int chunkId = docId / docsPerChunk;
chunkOffsetsWriter.writeInt(chunkId, newStartIndex);
}
customBitSet.setBit(newStartIndex);
prevRowStartIndex = newStartIndex;
prevRowLength = length;
- prevRowId = rowId;
return newStartIndex;
}
- @Override
- public void setCharArray(int row, char[] charArray) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setShortArray(int row, short[] shortsArray) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setIntArray(int row, int[] intArray) {
- rawDataWriter.writeInt(updateHeader(row, intArray.length), intArray.length, intArray);
- }
-
- @Override
- public void setLongArray(int row, long[] longArray) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setFloatArray(int row, float[] floatArray) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setDoubleArray(int row, double[] doubleArray) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setStringArray(int row, String[] stringArray) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setBytesArray(int row, byte[][] bytesArray) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
+ public void putDictIds(int[] dictIds) {
+ rawDataWriter.writeInt(updateHeader(dictIds.length), dictIds.length, dictIds);
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedBitSVForwardIndexWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedBitSVForwardIndexWriter.java
new file mode 100644
index 0000000..2bf94d6
--- /dev/null
+++ b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedBitSVForwardIndexWriter.java
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.io.writer.impl;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteOrder;
+import org.apache.pinot.core.io.util.FixedBitIntReaderWriter;
+import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+
+
+/**
+ * Bit-compressed dictionary-encoded forward index writer for single-value columns. The values written are dictionary
+ * ids.
+ */
+public class FixedBitSVForwardIndexWriter implements Closeable {
+ private final PinotDataBuffer _dataBuffer;
+ private final FixedBitIntReaderWriter _intReaderWriter;
+
+ private int _nextDocId = 0;
+
+ public FixedBitSVForwardIndexWriter(File file, int numDocs, int numBitsPerValue)
+ throws Exception {
+ // Convert to long in order to avoid int overflow
+ long length = ((long) numDocs * numBitsPerValue + Byte.SIZE - 1) / Byte.SIZE;
+ // Backward-compatible: index file is always big-endian
+ _dataBuffer = PinotDataBuffer.mapFile(file, false, 0, length, ByteOrder.BIG_ENDIAN, getClass().getSimpleName());
+ _intReaderWriter = new FixedBitIntReaderWriter(_dataBuffer, numDocs, numBitsPerValue);
+ }
+
+ public void putDictId(int dictId) {
+ _intReaderWriter.writeInt(_nextDocId++, dictId);
+ }
+
+ @Override
+ public void close()
+ throws IOException {
+ _intReaderWriter.close();
+ _dataBuffer.close();
+ }
+}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedByteChunkSingleValueWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedByteChunkSVForwardIndexWriter.java
similarity index 84%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedByteChunkSingleValueWriter.java
rename to pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedByteChunkSVForwardIndexWriter.java
index 7c83af8..2c9e64b 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedByteChunkSingleValueWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/FixedByteChunkSVForwardIndexWriter.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.writer.impl.v1;
+package org.apache.pinot.core.io.writer.impl;
import java.io.File;
import java.io.FileNotFoundException;
@@ -52,7 +52,7 @@ import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
* Only sequential writes are supported.
*/
@NotThreadSafe
-public class FixedByteChunkSingleValueWriter extends BaseChunkSingleValueWriter {
+public class FixedByteChunkSVForwardIndexWriter extends BaseChunkSVForwardIndexWriter {
private int _chunkDataOffset;
/**
@@ -66,7 +66,7 @@ public class FixedByteChunkSingleValueWriter extends BaseChunkSingleValueWriter
* @param writerVersion writer format version
* @throws FileNotFoundException Throws {@link FileNotFoundException} if the specified file is not found.
*/
- public FixedByteChunkSingleValueWriter(File file, ChunkCompressorFactory.CompressionType compressionType,
+ public FixedByteChunkSVForwardIndexWriter(File file, ChunkCompressorFactory.CompressionType compressionType,
int totalDocs, int numDocsPerChunk, int sizeOfEntry, int writerVersion)
throws FileNotFoundException {
super(file, compressionType, totalDocs, numDocsPerChunk, (sizeOfEntry * numDocsPerChunk), sizeOfEntry,
@@ -74,38 +74,27 @@ public class FixedByteChunkSingleValueWriter extends BaseChunkSingleValueWriter
_chunkDataOffset = 0;
}
- @Override
- public void setInt(int row, int value) {
+ public void putInt(int value) {
_chunkBuffer.putInt(value);
_chunkDataOffset += Integer.BYTES;
flushChunkIfNeeded();
}
- @Override
- public void setFloat(int row, float value) {
- _chunkBuffer.putFloat(value);
- _chunkDataOffset += Float.BYTES;
- flushChunkIfNeeded();
- }
-
- @Override
- public void setLong(int row, long value) {
+ public void putLong(long value) {
_chunkBuffer.putLong(value);
_chunkDataOffset += Long.BYTES;
flushChunkIfNeeded();
}
- @Override
- public void setDouble(int row, double value) {
- _chunkBuffer.putDouble(value);
- _chunkDataOffset += Double.BYTES;
+ public void putFloat(float value) {
+ _chunkBuffer.putFloat(value);
+ _chunkDataOffset += Float.BYTES;
flushChunkIfNeeded();
}
- @Override
- public void setBytes(int row, byte[] bytes) {
- _chunkBuffer.put(bytes);
- _chunkDataOffset += bytes.length;
+ public void putDouble(double value) {
+ _chunkBuffer.putDouble(value);
+ _chunkDataOffset += Double.BYTES;
flushChunkIfNeeded();
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/VarByteChunkSingleValueWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/VarByteChunkSVForwardIndexWriter.java
similarity index 88%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/VarByteChunkSingleValueWriter.java
rename to pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/VarByteChunkSVForwardIndexWriter.java
index 718544f..e2f0c5d 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/VarByteChunkSingleValueWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/VarByteChunkSVForwardIndexWriter.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.writer.impl.v1;
+package org.apache.pinot.core.io.writer.impl;
import java.io.File;
import java.io.FileNotFoundException;
@@ -52,7 +52,7 @@ import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
* Only sequential writes are supported.
*/
@NotThreadSafe
-public class VarByteChunkSingleValueWriter extends BaseChunkSingleValueWriter {
+public class VarByteChunkSVForwardIndexWriter extends BaseChunkSVForwardIndexWriter {
public static final int CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE = Integer.BYTES;
private final int _chunkHeaderSize;
@@ -70,8 +70,8 @@ public class VarByteChunkSingleValueWriter extends BaseChunkSingleValueWriter {
* @param writerVersion writer format version
* @throws FileNotFoundException Throws {@link FileNotFoundException} if the specified file is not found.
*/
- public VarByteChunkSingleValueWriter(File file, ChunkCompressorFactory.CompressionType compressionType, int totalDocs,
- int numDocsPerChunk, int lengthOfLongestEntry, int writerVersion)
+ public VarByteChunkSVForwardIndexWriter(File file, ChunkCompressorFactory.CompressionType compressionType,
+ int totalDocs, int numDocsPerChunk, int lengthOfLongestEntry, int writerVersion)
throws FileNotFoundException {
super(file, compressionType, totalDocs, numDocsPerChunk,
numDocsPerChunk * (CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE + lengthOfLongestEntry), // chunkSize
@@ -82,20 +82,17 @@ public class VarByteChunkSingleValueWriter extends BaseChunkSingleValueWriter {
_chunkDataOffSet = _chunkHeaderSize;
}
- @Override
- public void setString(int row, String string) {
- byte[] bytes = StringUtil.encodeUtf8(string);
- setBytes(row, bytes);
+ public void putString(String value) {
+ putBytes(StringUtil.encodeUtf8(value));
}
- @Override
- public void setBytes(int row, byte[] bytes) {
+ public void putBytes(byte[] value) {
_chunkBuffer.putInt(_chunkHeaderOffset, _chunkDataOffSet);
_chunkHeaderOffset += CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE;
_chunkBuffer.position(_chunkDataOffSet);
- _chunkBuffer.put(bytes);
- _chunkDataOffSet += bytes.length;
+ _chunkBuffer.put(value);
+ _chunkDataOffSet += value.length;
// If buffer filled, then compress and write to file.
if (_chunkHeaderOffset == _chunkHeaderSize) {
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedBitSingleValueWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedBitSingleValueWriter.java
deleted file mode 100644
index 7506293..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/impl/v1/FixedBitSingleValueWriter.java
+++ /dev/null
@@ -1,87 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.io.writer.impl.v1;
-
-import java.io.File;
-import java.io.IOException;
-import java.nio.ByteOrder;
-import org.apache.pinot.core.io.util.FixedBitIntReaderWriter;
-import org.apache.pinot.core.io.writer.SingleColumnSingleValueWriter;
-import org.apache.pinot.core.segment.memory.PinotDataBuffer;
-
-
-public class FixedBitSingleValueWriter implements SingleColumnSingleValueWriter {
- private FixedBitIntReaderWriter dataFileWriter;
-
- public FixedBitSingleValueWriter(File file, int rows, int columnSizeInBits)
- throws Exception {
- // Convert to long in order to avoid int overflow
- long length = ((long) rows * columnSizeInBits + Byte.SIZE - 1) / Byte.SIZE;
- // Backward-compatible: index file is always big-endian
- PinotDataBuffer dataBuffer =
- PinotDataBuffer.mapFile(file, false, 0, length, ByteOrder.BIG_ENDIAN, getClass().getSimpleName());
- dataFileWriter = new FixedBitIntReaderWriter(dataBuffer, rows, columnSizeInBits);
- }
-
- @Override
- public void close()
- throws IOException {
- dataFileWriter.close();
- }
-
- @Override
- public void setChar(int row, char ch) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setInt(int row, int i) {
- dataFileWriter.writeInt(row, i);
- }
-
- @Override
- public void setShort(int row, short s) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setLong(int row, long l) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setFloat(int row, float f) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setDouble(int row, double d) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setString(int row, String string) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-
- @Override
- public void setBytes(int row, byte[] bytes) {
- throw new UnsupportedOperationException("Only int data type is supported in fixedbit format");
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/minion/RawIndexConverter.java b/pinot-core/src/main/java/org/apache/pinot/core/minion/RawIndexConverter.java
index 0cfb813..ae7ebbc 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/minion/RawIndexConverter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/minion/RawIndexConverter.java
@@ -31,9 +31,8 @@ import org.apache.pinot.core.indexsegment.generator.SegmentVersion;
import org.apache.pinot.core.indexsegment.immutable.ImmutableSegment;
import org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentLoader;
import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
-import org.apache.pinot.core.io.writer.impl.v1.BaseChunkSingleValueWriter;
-import org.apache.pinot.core.operator.docvalsets.SingleValueSet;
-import org.apache.pinot.core.segment.creator.SingleValueRawIndexCreator;
+import org.apache.pinot.core.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.core.segment.creator.ForwardIndexCreator;
import org.apache.pinot.core.segment.creator.impl.SegmentColumnarIndexCreator;
import org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
@@ -41,6 +40,8 @@ import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.core.segment.index.readers.Dictionary;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
import org.apache.pinot.core.util.CrcUtils;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.MetricFieldSpec;
@@ -64,6 +65,7 @@ import org.slf4j.LoggerFactory;
* </ul>
* <p>After the conversion, add "rawIndex" into the segment metadata "optimizations" field.
*/
+@SuppressWarnings({"rawtypes", "unchecked"})
public class RawIndexConverter {
private static final Logger LOGGER = LoggerFactory.getLogger(RawIndexConverter.class);
@@ -196,16 +198,49 @@ public class RawIndexConverter {
// Create the raw index
DataSource dataSource = _originalImmutableSegment.getDataSource(columnName);
+ ForwardIndexReader reader = dataSource.getForwardIndex();
Dictionary dictionary = dataSource.getDictionary();
+ assert dictionary != null;
FieldSpec.DataType dataType = fieldSpec.getDataType();
int numDocs = _originalSegmentMetadata.getTotalDocs();
int lengthOfLongestEntry = _originalSegmentMetadata.getColumnMetadataFor(columnName).getColumnMaxLength();
- try (SingleValueRawIndexCreator rawIndexCreator = SegmentColumnarIndexCreator
+ try (ForwardIndexCreator rawIndexCreator = SegmentColumnarIndexCreator
.getRawIndexCreatorForColumn(_convertedIndexDir, ChunkCompressorFactory.CompressionType.SNAPPY, columnName,
- dataType, numDocs, lengthOfLongestEntry, false, BaseChunkSingleValueWriter.DEFAULT_VERSION)) {
- SingleValueSet valueSet = (SingleValueSet) dataSource.nextBlock().getBlockValueSet();
- for (int docId = 0; docId < numDocs; docId++) {
- rawIndexCreator.index(docId, dictionary.get(valueSet.getIntValue(docId)));
+ dataType, numDocs, lengthOfLongestEntry, false, BaseChunkSVForwardIndexWriter.DEFAULT_VERSION);
+ ForwardIndexReaderContext readerContext = reader.createContext()) {
+ switch (dataType) {
+ case INT:
+ for (int docId = 0; docId < numDocs; docId++) {
+ rawIndexCreator.putInt(dictionary.getIntValue(reader.getDictId(docId, readerContext)));
+ }
+ break;
+ case LONG:
+ for (int docId = 0; docId < numDocs; docId++) {
+ rawIndexCreator.putLong(dictionary.getLongValue(reader.getDictId(docId, readerContext)));
+ }
+ break;
+ case FLOAT:
+ for (int docId = 0; docId < numDocs; docId++) {
+ rawIndexCreator.putFloat(dictionary.getFloatValue(reader.getDictId(docId, readerContext)));
+ }
+ break;
+ case DOUBLE:
+ for (int docId = 0; docId < numDocs; docId++) {
+ rawIndexCreator.putDouble(dictionary.getDoubleValue(reader.getDictId(docId, readerContext)));
+ }
+ break;
+ case STRING:
+ for (int docId = 0; docId < numDocs; docId++) {
+ rawIndexCreator.putString(dictionary.getStringValue(reader.getDictId(docId, readerContext)));
+ }
+ break;
+ case BYTES:
+ for (int docId = 0; docId < numDocs; docId++) {
+ rawIndexCreator.putBytes(dictionary.getBytesValue(reader.getDictId(docId, readerContext)));
+ }
+ break;
+ default:
+ throw new IllegalStateException();
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/minion/segment/ReducerRecordReader.java b/pinot-core/src/main/java/org/apache/pinot/core/minion/segment/ReducerRecordReader.java
index 72a5fb8..8a14397 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/minion/segment/ReducerRecordReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/minion/segment/ReducerRecordReader.java
@@ -20,6 +20,7 @@ package org.apache.pinot.core.minion.segment;
import com.google.common.base.Preconditions;
import java.io.File;
+import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
@@ -118,7 +119,8 @@ public class ReducerRecordReader implements RecordReader {
}
@Override
- public void close() {
+ public void close()
+ throws IOException {
_pinotSegmentRecordReader.close();
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/BlockMetadataImpl.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/BlockMetadataImpl.java
deleted file mode 100644
index c0fac81..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/BlockMetadataImpl.java
+++ /dev/null
@@ -1,81 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.operator.blocks;
-
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.core.common.BlockMetadata;
-import org.apache.pinot.core.segment.index.readers.Dictionary;
-
-
-public final class BlockMetadataImpl implements BlockMetadata {
- private final int _numDocs;
- private final boolean _isSingleValue;
- private final int _maxNumMultiValues;
- private final FieldSpec.DataType _dataType;
- private final Dictionary _dictionary;
-
- public BlockMetadataImpl(int numDocs, boolean isSingleValue, int maxNumMultiValues, FieldSpec.DataType dataType,
- Dictionary dictionary) {
- _numDocs = numDocs;
- _isSingleValue = isSingleValue;
- _maxNumMultiValues = maxNumMultiValues;
- _dataType = dataType;
- _dictionary = dictionary;
- }
-
- @Override
- public int getLength() {
- return _numDocs;
- }
-
- @Override
- public int getStartDocId() {
- return 0;
- }
-
- @Override
- public int getEndDocId() {
- return _numDocs - 1;
- }
-
- @Override
- public FieldSpec.DataType getDataType() {
- return _dataType;
- }
-
- @Override
- public boolean isSingleValue() {
- return _isSingleValue;
- }
-
- @Override
- public int getMaxNumberOfMultiValues() {
- return _maxNumMultiValues;
- }
-
- @Override
- public boolean hasDictionary() {
- return _dictionary != null;
- }
-
- @Override
- public Dictionary getDictionary() {
- return _dictionary;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/MultiValueBlock.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/MultiValueBlock.java
deleted file mode 100644
index b07dc18..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/MultiValueBlock.java
+++ /dev/null
@@ -1,68 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.operator.blocks;
-
-import org.apache.pinot.core.common.Block;
-import org.apache.pinot.core.common.BlockDocIdSet;
-import org.apache.pinot.core.common.BlockDocIdValueSet;
-import org.apache.pinot.core.common.BlockMetadata;
-import org.apache.pinot.core.common.BlockValSet;
-import org.apache.pinot.core.io.reader.SingleColumnMultiValueReader;
-import org.apache.pinot.core.operator.docvalsets.MultiValueSet;
-import org.apache.pinot.core.segment.index.readers.Dictionary;
-import org.apache.pinot.spi.data.FieldSpec;
-
-
-@SuppressWarnings("rawtypes")
-public final class MultiValueBlock implements Block {
- private final SingleColumnMultiValueReader _reader;
- private final FieldSpec.DataType _dataType;
- private final BlockMetadata _blockMetadata;
- private BlockValSet _blockValSet;
-
- public MultiValueBlock(SingleColumnMultiValueReader reader, int numDocs, int maxNumMultiValues,
- FieldSpec.DataType dataType, Dictionary dictionary) {
- _reader = reader;
- _dataType = dataType;
- _blockMetadata = new BlockMetadataImpl(numDocs, false, maxNumMultiValues, dataType, dictionary);
- }
-
- @Override
- public BlockDocIdSet getBlockDocIdSet() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public BlockValSet getBlockValueSet() {
- if (_blockValSet == null) {
- _blockValSet = new MultiValueSet(_reader, _dataType);
- }
- return _blockValSet;
- }
-
- @Override
- public BlockDocIdValueSet getBlockDocIdValueSet() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public BlockMetadata getMetadata() {
- return _blockMetadata;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/SingleValueBlock.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/SingleValueBlock.java
deleted file mode 100644
index 6e9615b..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/SingleValueBlock.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.operator.blocks;
-
-import org.apache.pinot.core.common.Block;
-import org.apache.pinot.core.common.BlockDocIdSet;
-import org.apache.pinot.core.common.BlockDocIdValueSet;
-import org.apache.pinot.core.common.BlockMetadata;
-import org.apache.pinot.core.common.BlockValSet;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.core.operator.docvalsets.SingleValueSet;
-import org.apache.pinot.core.segment.index.readers.Dictionary;
-import org.apache.pinot.spi.data.FieldSpec;
-
-
-@SuppressWarnings("rawtypes")
-public final class SingleValueBlock implements Block {
- private final SingleColumnSingleValueReader _reader;
- private final FieldSpec.DataType _dataType;
- private final BlockMetadata _blockMetadata;
- private BlockValSet _blockValSet;
-
- public SingleValueBlock(SingleColumnSingleValueReader reader, int numDocs, FieldSpec.DataType dataType,
- Dictionary dictionary) {
- _reader = reader;
- _dataType = dataType;
- _blockMetadata = new BlockMetadataImpl(numDocs, true, 0, dataType, dictionary);
- }
-
- public SingleColumnSingleValueReader getReader() {
- return _reader;
- }
-
- @Override
- public BlockDocIdSet getBlockDocIdSet() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public BlockValSet getBlockValueSet() {
- if (_blockValSet == null) {
- _blockValSet = new SingleValueSet(_reader, _dataType);
- }
- return _blockValSet;
- }
-
- @Override
- public BlockDocIdValueSet getBlockDocIdValueSet() {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public BlockMetadata getMetadata() {
- return _blockMetadata;
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/MVScanDocIdIterator.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/MVScanDocIdIterator.java
index 8d6cbb7..c41ccbe 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/MVScanDocIdIterator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/MVScanDocIdIterator.java
@@ -19,8 +19,9 @@
package org.apache.pinot.core.operator.dociditerators;
import org.apache.pinot.core.common.Constants;
-import org.apache.pinot.core.operator.docvalsets.MultiValueSet;
import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
import org.roaringbitmap.IntIterator;
import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
@@ -30,19 +31,23 @@ import org.roaringbitmap.buffer.MutableRoaringBitmap;
* The {@code MVScanDocIdIterator} is the scan-based iterator for MVScanDocIdSet to scan a multi-value column for the
* matching document ids.
*/
+@SuppressWarnings({"rawtypes", "unchecked"})
public final class MVScanDocIdIterator implements ScanBasedDocIdIterator {
private final PredicateEvaluator _predicateEvaluator;
- private final MultiValueSet _valueSet;
+ private final ForwardIndexReader _reader;
+ // TODO: Figure out a way to close the reader context
+ private final ForwardIndexReaderContext _readerContext;
private final int _numDocs;
private final int[] _dictIdBuffer;
private int _nextDocId = 0;
private long _numEntriesScanned = 0L;
- public MVScanDocIdIterator(PredicateEvaluator predicateEvaluator, MultiValueSet valueSet, int numDocs,
+ public MVScanDocIdIterator(PredicateEvaluator predicateEvaluator, ForwardIndexReader reader, int numDocs,
int maxNumEntriesPerValue) {
_predicateEvaluator = predicateEvaluator;
- _valueSet = valueSet;
+ _reader = reader;
+ _readerContext = reader.createContext();
_numDocs = numDocs;
_dictIdBuffer = new int[maxNumEntriesPerValue];
}
@@ -51,7 +56,7 @@ public final class MVScanDocIdIterator implements ScanBasedDocIdIterator {
public int next() {
while (_nextDocId < _numDocs) {
int nextDocId = _nextDocId++;
- int length = _valueSet.getIntValues(nextDocId, _dictIdBuffer);
+ int length = _reader.getDictIdMV(nextDocId, _dictIdBuffer, _readerContext);
_numEntriesScanned += length;
if (_predicateEvaluator.applyMV(_dictIdBuffer, length)) {
return nextDocId;
@@ -72,7 +77,7 @@ public final class MVScanDocIdIterator implements ScanBasedDocIdIterator {
IntIterator docIdIterator = docIds.getIntIterator();
int nextDocId;
while (docIdIterator.hasNext() && (nextDocId = docIdIterator.next()) < _numDocs) {
- int length = _valueSet.getIntValues(nextDocId, _dictIdBuffer);
+ int length = _reader.getDictIdMV(nextDocId, _dictIdBuffer, _readerContext);
_numEntriesScanned += length;
if (_predicateEvaluator.applyMV(_dictIdBuffer, length)) {
result.add(nextDocId);
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java
index bf8d36d..b7bc878 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java
@@ -19,8 +19,9 @@
package org.apache.pinot.core.operator.dociditerators;
import org.apache.pinot.core.common.Constants;
-import org.apache.pinot.core.operator.docvalsets.SingleValueSet;
import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
import org.roaringbitmap.IntIterator;
import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
@@ -30,18 +31,23 @@ import org.roaringbitmap.buffer.MutableRoaringBitmap;
* The {@code SVScanDocIdIterator} is the scan-based iterator for SVScanDocIdSet to scan a single-value column for the
* matching document ids.
*/
+@SuppressWarnings({"rawtypes", "unchecked"})
public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
private final PredicateEvaluator _predicateEvaluator;
- private final SingleValueSet _valueSet;
+ private final ForwardIndexReader _reader;
+ // TODO: Figure out a way to close the reader context
+ // ChunkReaderContext should be closed explicitly to release the off-heap buffer
+ private final ForwardIndexReaderContext _readerContext;
private final int _numDocs;
private final ValueMatcher _valueMatcher;
private int _nextDocId = 0;
private long _numEntriesScanned = 0L;
- public SVScanDocIdIterator(PredicateEvaluator predicateEvaluator, SingleValueSet valueSet, int numDocs) {
+ public SVScanDocIdIterator(PredicateEvaluator predicateEvaluator, ForwardIndexReader reader, int numDocs) {
_predicateEvaluator = predicateEvaluator;
- _valueSet = valueSet;
+ _reader = reader;
+ _readerContext = reader.createContext();
_numDocs = numDocs;
_valueMatcher = getValueMatcher();
}
@@ -84,21 +90,25 @@ public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
}
private ValueMatcher getValueMatcher() {
- switch (_predicateEvaluator.getDataType()) {
- case INT:
- return new IntMatcher();
- case LONG:
- return new LongMatcher();
- case FLOAT:
- return new FloatMatcher();
- case DOUBLE:
- return new DoubleMatcher();
- case STRING:
- return new StringMatcher();
- case BYTES:
- return new BytesMatcher();
- default:
- throw new UnsupportedOperationException();
+ if (_reader.isDictionaryEncoded()) {
+ return new DictIdMatcher();
+ } else {
+ switch (_reader.getValueType()) {
+ case INT:
+ return new IntMatcher();
+ case LONG:
+ return new LongMatcher();
+ case FLOAT:
+ return new FloatMatcher();
+ case DOUBLE:
+ return new DoubleMatcher();
+ case STRING:
+ return new StringMatcher();
+ case BYTES:
+ return new BytesMatcher();
+ default:
+ throw new UnsupportedOperationException();
+ }
}
}
@@ -110,11 +120,19 @@ public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
boolean doesValueMatch(int docId);
}
+ private class DictIdMatcher implements ValueMatcher {
+
+ @Override
+ public boolean doesValueMatch(int docId) {
+ return _predicateEvaluator.applySV(_reader.getDictId(docId, _readerContext));
+ }
+ }
+
private class IntMatcher implements ValueMatcher {
@Override
public boolean doesValueMatch(int docId) {
- return _predicateEvaluator.applySV(_valueSet.getIntValue(docId));
+ return _predicateEvaluator.applySV(_reader.getInt(docId, _readerContext));
}
}
@@ -122,7 +140,7 @@ public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
@Override
public boolean doesValueMatch(int docId) {
- return _predicateEvaluator.applySV(_valueSet.getLongValue(docId));
+ return _predicateEvaluator.applySV(_reader.getLong(docId, _readerContext));
}
}
@@ -130,7 +148,7 @@ public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
@Override
public boolean doesValueMatch(int docId) {
- return _predicateEvaluator.applySV(_valueSet.getFloatValue(docId));
+ return _predicateEvaluator.applySV(_reader.getFloat(docId, _readerContext));
}
}
@@ -138,7 +156,7 @@ public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
@Override
public boolean doesValueMatch(int docId) {
- return _predicateEvaluator.applySV(_valueSet.getDoubleValue(docId));
+ return _predicateEvaluator.applySV(_reader.getDouble(docId, _readerContext));
}
}
@@ -146,7 +164,7 @@ public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
@Override
public boolean doesValueMatch(int docId) {
- return _predicateEvaluator.applySV(_valueSet.getStringValue(docId));
+ return _predicateEvaluator.applySV(_reader.getString(docId, _readerContext));
}
}
@@ -154,7 +172,7 @@ public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
@Override
public boolean doesValueMatch(int docId) {
- return _predicateEvaluator.applySV(_valueSet.getBytesValue(docId));
+ return _predicateEvaluator.applySV(_reader.getBytes(docId, _readerContext));
}
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/MVScanDocIdSet.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/MVScanDocIdSet.java
index a2baa67..ad50b11 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/MVScanDocIdSet.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/MVScanDocIdSet.java
@@ -19,16 +19,16 @@
package org.apache.pinot.core.operator.docidsets;
import org.apache.pinot.core.operator.dociditerators.MVScanDocIdIterator;
-import org.apache.pinot.core.operator.docvalsets.MultiValueSet;
import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
public final class MVScanDocIdSet implements FilterBlockDocIdSet {
private final MVScanDocIdIterator _docIdIterator;
- public MVScanDocIdSet(PredicateEvaluator predicateEvaluator, MultiValueSet valueSet, int numDocs,
+ public MVScanDocIdSet(PredicateEvaluator predicateEvaluator, ForwardIndexReader<?> reader, int numDocs,
int maxNumEntriesPerValue) {
- _docIdIterator = new MVScanDocIdIterator(predicateEvaluator, valueSet, numDocs, maxNumEntriesPerValue);
+ _docIdIterator = new MVScanDocIdIterator(predicateEvaluator, reader, numDocs, maxNumEntriesPerValue);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/SVScanDocIdSet.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/SVScanDocIdSet.java
index 2fb3e59..88a3a35 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/SVScanDocIdSet.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/SVScanDocIdSet.java
@@ -19,15 +19,15 @@
package org.apache.pinot.core.operator.docidsets;
import org.apache.pinot.core.operator.dociditerators.SVScanDocIdIterator;
-import org.apache.pinot.core.operator.docvalsets.SingleValueSet;
import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
public final class SVScanDocIdSet implements FilterBlockDocIdSet {
private final SVScanDocIdIterator _docIdIterator;
- public SVScanDocIdSet(PredicateEvaluator predicateEvaluator, SingleValueSet valueSet, int numDocs) {
- _docIdIterator = new SVScanDocIdIterator(predicateEvaluator, valueSet, numDocs);
+ public SVScanDocIdSet(PredicateEvaluator predicateEvaluator, ForwardIndexReader<?> reader, int numDocs) {
+ _docIdIterator = new SVScanDocIdIterator(predicateEvaluator, reader, numDocs);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/ProjectionBlockValSet.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/ProjectionBlockValSet.java
index 98d4cfd..efdb41b 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/ProjectionBlockValSet.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/ProjectionBlockValSet.java
@@ -18,10 +18,10 @@
*/
package org.apache.pinot.core.operator.docvalsets;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-import org.apache.pinot.core.common.BaseBlockValSet;
+import org.apache.pinot.core.common.BlockValSet;
import org.apache.pinot.core.common.DataBlockCache;
import org.apache.pinot.core.operator.ProjectionOperator;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
/**
@@ -29,7 +29,7 @@ import org.apache.pinot.core.operator.ProjectionOperator;
* It provides api's to access data for a specified projection column.
* It uses {@link DataBlockCache} to cache the projection data.
*/
-public class ProjectionBlockValSet extends BaseBlockValSet {
+public class ProjectionBlockValSet implements BlockValSet {
private final DataBlockCache _dataBlockCache;
private final String _column;
private final DataType _dataType;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/SingleValueSet.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/SingleValueSet.java
deleted file mode 100644
index 9a649bc..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/SingleValueSet.java
+++ /dev/null
@@ -1,185 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.operator.docvalsets;
-
-import org.apache.pinot.core.common.BaseBlockValSet;
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.spi.data.FieldSpec.DataType;
-
-
-@SuppressWarnings({"rawtypes", "unchecked"})
-public final class SingleValueSet extends BaseBlockValSet {
- private final SingleColumnSingleValueReader _reader;
- private final ReaderContext _readerContext;
- private final DataType _dataType;
-
- public SingleValueSet(SingleColumnSingleValueReader reader, DataType dataType) {
- _reader = reader;
- _readerContext = reader.createContext();
- _dataType = dataType;
- }
-
- @Override
- public DataType getValueType() {
- return _dataType;
- }
-
- @Override
- public boolean isSingleValue() {
- return true;
- }
-
- @Override
- public int getIntValue(int docId) {
- return _reader.getInt(docId, _readerContext);
- }
-
- @Override
- public long getLongValue(int docId) {
- return _reader.getLong(docId, _readerContext);
- }
-
- @Override
- public float getFloatValue(int docId) {
- return _reader.getFloat(docId, _readerContext);
- }
-
- @Override
- public double getDoubleValue(int docId) {
- return _reader.getDouble(docId, _readerContext);
- }
-
- @Override
- public String getStringValue(int docId) {
- return _reader.getString(docId, _readerContext);
- }
-
- @Override
- public byte[] getBytesValue(int docId) {
- return _reader.getBytes(docId, _readerContext);
- }
-
- @Override
- public void getDictionaryIds(int[] docIds, int length, int[] dictIdBuffer) {
- _reader.readValues(docIds, 0, length, dictIdBuffer, 0);
- }
-
- @Override
- public void getIntValues(int[] docIds, int length, int[] valueBuffer) {
- if (_dataType == DataType.INT) {
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getInt(docIds[i], _readerContext);
- }
- } else {
- throw new IllegalStateException(String.format("Cannot read %s as INT", _dataType));
- }
- }
-
- @Override
- public void getLongValues(int[] docIds, int length, long[] valueBuffer) {
- switch (_dataType) {
- case INT:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getInt(docIds[i], _readerContext);
- }
- break;
- case LONG:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getLong(docIds[i], _readerContext);
- }
- break;
- default:
- throw new IllegalStateException(String.format("Cannot read %s as LONG", _dataType));
- }
- }
-
- @Override
- public void getFloatValues(int[] docIds, int length, float[] valueBuffer) {
- switch (_dataType) {
- case INT:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getInt(docIds[i], _readerContext);
- }
- break;
- case LONG:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getLong(docIds[i], _readerContext);
- }
- break;
- case FLOAT:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getFloat(docIds[i], _readerContext);
- }
- break;
- default:
- throw new IllegalStateException(String.format("Cannot read %s as FLOAT", _dataType));
- }
- }
-
- @Override
- public void getDoubleValues(int[] docIds, int length, double[] valueBuffer) {
- switch (_dataType) {
- case INT:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getInt(docIds[i], _readerContext);
- }
- break;
- case LONG:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getLong(docIds[i], _readerContext);
- }
- break;
- case FLOAT:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getFloat(docIds[i], _readerContext);
- }
- break;
- case DOUBLE:
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getDouble(docIds[i], _readerContext);
- }
- break;
- default:
- throw new IllegalStateException(String.format("Cannot read %s as DOUBLE", _dataType));
- }
- }
-
- @Override
- public void getStringValues(int[] docIds, int length, String[] valueBuffer) {
- if (_dataType == DataType.STRING) {
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getString(docIds[i], _readerContext);
- }
- } else {
- throw new IllegalStateException(String.format("Cannot read %s as STRING", _dataType));
- }
- }
-
- @Override
- public void getBytesValues(int[] docIds, int length, byte[][] valueBuffer) {
- if (_dataType.equals(DataType.BYTES)) {
- for (int i = 0; i < length; i++) {
- valueBuffer[i] = _reader.getBytes(docIds[i], _readerContext);
- }
- } else {
- throw new IllegalStateException(String.format("Cannot read %s as BYTES", _dataType));
- }
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/TransformBlockValSet.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/TransformBlockValSet.java
index 1a5262f..78fa38a 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/TransformBlockValSet.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/TransformBlockValSet.java
@@ -18,12 +18,12 @@
*/
package org.apache.pinot.core.operator.docvalsets;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.core.common.BaseBlockValSet;
+import org.apache.pinot.core.common.BlockValSet;
import org.apache.pinot.core.operator.blocks.ProjectionBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
import org.apache.pinot.core.operator.transform.function.TransformFunction;
import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.spi.data.FieldSpec;
/**
@@ -31,7 +31,7 @@ import org.apache.pinot.core.plan.DocIdSetPlanNode;
* block.
* <p>Caller is responsible for calling the correct method based on the data source metadata for the block value set.
*/
-public class TransformBlockValSet extends BaseBlockValSet {
+public class TransformBlockValSet implements BlockValSet {
private final ProjectionBlock _projectionBlock;
private final TransformFunction _transformFunction;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/ScanBasedFilterOperator.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/ScanBasedFilterOperator.java
index 621523f..aaf8a93 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/ScanBasedFilterOperator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/ScanBasedFilterOperator.java
@@ -18,15 +18,11 @@
*/
package org.apache.pinot.core.operator.filter;
-import org.apache.pinot.core.common.BlockValSet;
import org.apache.pinot.core.common.DataSource;
import org.apache.pinot.core.common.DataSourceMetadata;
import org.apache.pinot.core.operator.blocks.FilterBlock;
-import org.apache.pinot.core.operator.docidsets.FilterBlockDocIdSet;
import org.apache.pinot.core.operator.docidsets.MVScanDocIdSet;
import org.apache.pinot.core.operator.docidsets.SVScanDocIdSet;
-import org.apache.pinot.core.operator.docvalsets.MultiValueSet;
-import org.apache.pinot.core.operator.docvalsets.SingleValueSet;
import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
@@ -46,16 +42,12 @@ public class ScanBasedFilterOperator extends BaseFilterOperator {
@Override
protected FilterBlock getNextBlock() {
DataSourceMetadata dataSourceMetadata = _dataSource.getDataSourceMetadata();
- BlockValSet valueSet = _dataSource.nextBlock().getBlockValueSet();
-
- FilterBlockDocIdSet docIdSet;
if (dataSourceMetadata.isSingleValue()) {
- docIdSet = new SVScanDocIdSet(_predicateEvaluator, (SingleValueSet) valueSet, _numDocs);
+ return new FilterBlock(new SVScanDocIdSet(_predicateEvaluator, _dataSource.getForwardIndex(), _numDocs));
} else {
- docIdSet = new MVScanDocIdSet(_predicateEvaluator, (MultiValueSet) valueSet, _numDocs,
- dataSourceMetadata.getMaxNumValuesPerMVEntry());
+ return new FilterBlock(new MVScanDocIdSet(_predicateEvaluator, _dataSource.getForwardIndex(), _numDocs,
+ dataSourceMetadata.getMaxNumValuesPerMVEntry()));
}
- return new FilterBlock(docIdSet);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/SortedIndexBasedFilterOperator.java b/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/SortedIndexBasedFilterOperator.java
index 2fa417c..4dc25ee 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/SortedIndexBasedFilterOperator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/SortedIndexBasedFilterOperator.java
@@ -25,24 +25,23 @@ import java.util.Collections;
import java.util.List;
import org.apache.pinot.common.utils.Pairs.IntPair;
import org.apache.pinot.core.common.DataSource;
-import org.apache.pinot.core.io.reader.impl.v1.SortedIndexReader;
+import org.apache.pinot.core.segment.index.readers.SortedIndexReader;
import org.apache.pinot.core.operator.blocks.FilterBlock;
import org.apache.pinot.core.operator.docidsets.SortedDocIdSet;
import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
import org.apache.pinot.core.operator.filter.predicate.RangePredicateEvaluatorFactory.OfflineDictionaryBasedRangePredicateEvaluator;
-@SuppressWarnings("rawtypes")
public class SortedIndexBasedFilterOperator extends BaseFilterOperator {
private static final String OPERATOR_NAME = "SortedIndexBasedFilterOperator";
private final PredicateEvaluator _predicateEvaluator;
- private final SortedIndexReader _sortedIndexReader;
+ private final SortedIndexReader<?> _sortedIndexReader;
private final int _numDocs;
SortedIndexBasedFilterOperator(PredicateEvaluator predicateEvaluator, DataSource dataSource, int numDocs) {
_predicateEvaluator = predicateEvaluator;
- _sortedIndexReader = (SortedIndexReader) dataSource.getInvertedIndex();
+ _sortedIndexReader = (SortedIndexReader<?>) dataSource.getInvertedIndex();
_numDocs = numDocs;
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/stats/RealtimeColumnStatistics.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/stats/RealtimeColumnStatistics.java
index 4f91377..7970f1a 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/stats/RealtimeColumnStatistics.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/stats/RealtimeColumnStatistics.java
@@ -19,14 +19,12 @@
package org.apache.pinot.core.realtime.converter.stats;
import java.util.Set;
-import org.apache.pinot.core.common.Block;
-import org.apache.pinot.core.common.BlockMetadata;
import org.apache.pinot.core.common.DataSource;
+import org.apache.pinot.core.common.DataSourceMetadata;
import org.apache.pinot.core.data.partition.PartitionFunction;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.core.operator.blocks.SingleValueBlock;
import org.apache.pinot.core.realtime.impl.dictionary.BaseMutableDictionary;
import org.apache.pinot.core.segment.creator.ColumnStatistics;
+import org.apache.pinot.core.segment.index.readers.MutableForwardIndex;
import org.apache.pinot.spi.data.FieldSpec;
@@ -112,11 +110,10 @@ public class RealtimeColumnStatistics implements ColumnStatistics {
@Override
public boolean isSorted() {
- Block block = _dataSource.nextBlock();
- BlockMetadata blockMetadata = block.getMetadata();
+ DataSourceMetadata dataSourceMetadata = _dataSource.getDataSourceMetadata();
// Multi-valued column cannot be sorted
- if (!blockMetadata.isSingleValue()) {
+ if (!dataSourceMetadata.isSingleValue()) {
return false;
}
@@ -126,23 +123,22 @@ public class RealtimeColumnStatistics implements ColumnStatistics {
}
// Iterate over all data to figure out whether or not it's in sorted order
- SingleColumnSingleValueReader singleValueReader = ((SingleValueBlock) block).getReader();
-
- int numDocs = blockMetadata.getLength();
+ MutableForwardIndex mutableForwardIndex = (MutableForwardIndex) _dataSource.getForwardIndex();
+ int numDocs = dataSourceMetadata.getNumDocs();
// Iterate with the sorted order if provided
if (_sortedDocIdIterationOrder != null) {
- int previousDictId = singleValueReader.getInt(_sortedDocIdIterationOrder[0]);
+ int previousDictId = mutableForwardIndex.getDictId(_sortedDocIdIterationOrder[0]);
for (int i = 1; i < numDocs; i++) {
- int currentDictId = singleValueReader.getInt(_sortedDocIdIterationOrder[i]);
+ int currentDictId = mutableForwardIndex.getDictId(_sortedDocIdIterationOrder[i]);
if (_mutableDictionary.compare(previousDictId, currentDictId) > 0) {
return false;
}
previousDictId = currentDictId;
}
} else {
- int previousDictId = singleValueReader.getInt(0);
+ int previousDictId = mutableForwardIndex.getDictId(0);
for (int i = 1; i < numDocs; i++) {
- int currentDictId = singleValueReader.getInt(i);
+ int currentDictId = mutableForwardIndex.getDictId(i);
if (_mutableDictionary.compare(previousDictId, currentDictId) > 0) {
return false;
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/stats/RealtimeNoDictionaryColStatistics.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/stats/RealtimeNoDictionaryColStatistics.java
index 7e49504..d639b17 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/stats/RealtimeNoDictionaryColStatistics.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/stats/RealtimeNoDictionaryColStatistics.java
@@ -22,7 +22,7 @@ import java.util.Set;
import org.apache.pinot.core.common.DataSource;
import org.apache.pinot.core.common.DataSourceMetadata;
import org.apache.pinot.core.data.partition.PartitionFunction;
-import org.apache.pinot.core.io.readerwriter.BaseSingleColumnSingleValueReaderWriter;
+import org.apache.pinot.core.segment.index.readers.MutableForwardIndex;
import org.apache.pinot.core.segment.creator.ColumnStatistics;
import static org.apache.pinot.core.common.Constants.UNKNOWN_CARDINALITY;
@@ -30,11 +30,11 @@ import static org.apache.pinot.core.common.Constants.UNKNOWN_CARDINALITY;
public class RealtimeNoDictionaryColStatistics implements ColumnStatistics {
private final DataSourceMetadata _dataSourceMetadata;
- private final BaseSingleColumnSingleValueReaderWriter _forwardIndex;
+ private final MutableForwardIndex _forwardIndex;
public RealtimeNoDictionaryColStatistics(DataSource dataSource) {
_dataSourceMetadata = dataSource.getDataSourceMetadata();
- _forwardIndex = (BaseSingleColumnSingleValueReaderWriter) dataSource.getForwardIndex();
+ _forwardIndex = (MutableForwardIndex) dataSource.getForwardIndex();
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/DoubleOffHeapMutableDictionary.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/DoubleOffHeapMutableDictionary.java
index 3cfd5d0..b1020d5 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/DoubleOffHeapMutableDictionary.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/DoubleOffHeapMutableDictionary.java
@@ -24,13 +24,14 @@ import it.unimi.dsi.fastutil.ints.IntSets;
import java.io.IOException;
import java.util.Arrays;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
-import org.apache.pinot.core.io.readerwriter.impl.FixedByteSingleColumnSingleValueReaderWriter;
import org.apache.pinot.core.query.request.context.predicate.RangePredicate;
+import org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
@SuppressWarnings("Duplicates")
public class DoubleOffHeapMutableDictionary extends BaseOffHeapMutableDictionary {
- private final FixedByteSingleColumnSingleValueReaderWriter _dictIdToValue;
+ private final FixedByteSVMutableForwardIndex _dictIdToValue;
private volatile double _min = Double.MAX_VALUE;
private volatile double _max = Double.MIN_VALUE;
@@ -39,8 +40,8 @@ public class DoubleOffHeapMutableDictionary extends BaseOffHeapMutableDictionary
PinotDataBufferMemoryManager memoryManager, String allocationContext) {
super(estimatedCardinality, maxOverflowSize, memoryManager, allocationContext);
int initialEntryCount = nearestPowerOf2(estimatedCardinality);
- _dictIdToValue = new FixedByteSingleColumnSingleValueReaderWriter(initialEntryCount, Double.BYTES, memoryManager,
- allocationContext);
+ _dictIdToValue =
+ new FixedByteSVMutableForwardIndex(false, DataType.DOUBLE, initialEntryCount, memoryManager, allocationContext);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/FloatOffHeapMutableDictionary.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/FloatOffHeapMutableDictionary.java
index 2482c84..4ceb606 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/FloatOffHeapMutableDictionary.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/FloatOffHeapMutableDictionary.java
@@ -24,13 +24,14 @@ import it.unimi.dsi.fastutil.ints.IntSets;
import java.io.IOException;
import java.util.Arrays;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
-import org.apache.pinot.core.io.readerwriter.impl.FixedByteSingleColumnSingleValueReaderWriter;
import org.apache.pinot.core.query.request.context.predicate.RangePredicate;
+import org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
@SuppressWarnings("Duplicates")
public class FloatOffHeapMutableDictionary extends BaseOffHeapMutableDictionary {
- private final FixedByteSingleColumnSingleValueReaderWriter _dictIdToValue;
+ private final FixedByteSVMutableForwardIndex _dictIdToValue;
private volatile float _min = Float.MAX_VALUE;
private volatile float _max = Float.MIN_VALUE;
@@ -39,8 +40,8 @@ public class FloatOffHeapMutableDictionary extends BaseOffHeapMutableDictionary
PinotDataBufferMemoryManager memoryManager, String allocationContext) {
super(estimatedCardinality, maxOverflowSize, memoryManager, allocationContext);
int initialEntryCount = nearestPowerOf2(estimatedCardinality);
- _dictIdToValue = new FixedByteSingleColumnSingleValueReaderWriter(initialEntryCount, Float.BYTES, memoryManager,
- allocationContext);
+ _dictIdToValue =
+ new FixedByteSVMutableForwardIndex(false, DataType.FLOAT, initialEntryCount, memoryManager, allocationContext);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/IntOffHeapMutableDictionary.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/IntOffHeapMutableDictionary.java
index 58ddc7e..c5d5e40 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/IntOffHeapMutableDictionary.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/IntOffHeapMutableDictionary.java
@@ -24,13 +24,14 @@ import it.unimi.dsi.fastutil.ints.IntSets;
import java.io.IOException;
import java.util.Arrays;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
-import org.apache.pinot.core.io.readerwriter.impl.FixedByteSingleColumnSingleValueReaderWriter;
import org.apache.pinot.core.query.request.context.predicate.RangePredicate;
+import org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
@SuppressWarnings("Duplicates")
public class IntOffHeapMutableDictionary extends BaseOffHeapMutableDictionary {
- private final FixedByteSingleColumnSingleValueReaderWriter _dictIdToValue;
+ private final FixedByteSVMutableForwardIndex _dictIdToValue;
private volatile int _min = Integer.MAX_VALUE;
private volatile int _max = Integer.MIN_VALUE;
@@ -39,8 +40,8 @@ public class IntOffHeapMutableDictionary extends BaseOffHeapMutableDictionary {
PinotDataBufferMemoryManager memoryManager, String allocationContext) {
super(estimatedCardinality, maxOverflowSize, memoryManager, allocationContext);
int initialEntryCount = nearestPowerOf2(estimatedCardinality);
- _dictIdToValue = new FixedByteSingleColumnSingleValueReaderWriter(initialEntryCount, Integer.BYTES, memoryManager,
- allocationContext);
+ _dictIdToValue =
+ new FixedByteSVMutableForwardIndex(false, DataType.INT, initialEntryCount, memoryManager, allocationContext);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/LongOffHeapMutableDictionary.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/LongOffHeapMutableDictionary.java
index fd1f305..b1d62c7 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/LongOffHeapMutableDictionary.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/dictionary/LongOffHeapMutableDictionary.java
@@ -24,13 +24,14 @@ import it.unimi.dsi.fastutil.ints.IntSets;
import java.io.IOException;
import java.util.Arrays;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
-import org.apache.pinot.core.io.readerwriter.impl.FixedByteSingleColumnSingleValueReaderWriter;
import org.apache.pinot.core.query.request.context.predicate.RangePredicate;
+import org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
@SuppressWarnings("Duplicates")
public class LongOffHeapMutableDictionary extends BaseOffHeapMutableDictionary {
- private final FixedByteSingleColumnSingleValueReaderWriter _dictIdToValue;
+ private final FixedByteSVMutableForwardIndex _dictIdToValue;
private volatile long _min = Long.MAX_VALUE;
private volatile long _max = Long.MIN_VALUE;
@@ -39,8 +40,8 @@ public class LongOffHeapMutableDictionary extends BaseOffHeapMutableDictionary {
PinotDataBufferMemoryManager memoryManager, String allocationContext) {
super(estimatedCardinality, overflowSize, memoryManager, allocationContext);
int initialEntryCount = nearestPowerOf2(estimatedCardinality);
- _dictIdToValue = new FixedByteSingleColumnSingleValueReaderWriter(initialEntryCount, Long.BYTES, memoryManager,
- allocationContext);
+ _dictIdToValue =
+ new FixedByteSVMutableForwardIndex(false, DataType.LONG, initialEntryCount, memoryManager, allocationContext);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleColumnMultiValueReaderWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/FixedByteMVMutableForwardIndex.java
similarity index 66%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleColumnMultiValueReaderWriter.java
rename to pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/FixedByteMVMutableForwardIndex.java
index b47cfb7..fe0e318 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleColumnMultiValueReaderWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/FixedByteMVMutableForwardIndex.java
@@ -16,16 +16,17 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.readerwriter.impl;
+package org.apache.pinot.core.realtime.impl.forward;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.pinot.core.io.reader.impl.FixedByteSingleValueMultiColReader;
-import org.apache.pinot.core.io.readerwriter.BaseSingleColumnMultiValueReaderWriter;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
import org.apache.pinot.core.io.writer.impl.FixedByteSingleValueMultiColWriter;
+import org.apache.pinot.core.segment.index.readers.MutableForwardIndex;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -86,10 +87,11 @@ import org.slf4j.LoggerFactory;
* </code>
*
*/
-// TODO: Check thread-safety
-public class FixedByteSingleColumnMultiValueReaderWriter extends BaseSingleColumnMultiValueReaderWriter {
+// TODO: Fix thread-safety issue for ArrayList
+// TODO: Optimize it
+public class FixedByteMVMutableForwardIndex implements MutableForwardIndex {
+ private static final Logger LOGGER = LoggerFactory.getLogger(FixedByteMVMutableForwardIndex.class);
- private static final Logger LOGGER = LoggerFactory.getLogger(FixedByteSingleColumnMultiValueReaderWriter.class);
/**
* number of columns is 1, column size is variable but less than _maxNumberOfMultiValuesPerRow
*/
@@ -118,8 +120,8 @@ public class FixedByteSingleColumnMultiValueReaderWriter extends BaseSingleColum
private int _prevRowStartIndex = 0; // Offset in the data-buffer for the last row added.
private int _prevRowLength = 0; // Number of values in the column for the last row added.
- public FixedByteSingleColumnMultiValueReaderWriter(int maxNumberOfMultiValuesPerRow, int avgMultiValueCount,
- int rowCountPerChunk, int columnSizeInBytes, PinotDataBufferMemoryManager memoryManager, String context) {
+ public FixedByteMVMutableForwardIndex(int maxNumberOfMultiValuesPerRow, int avgMultiValueCount, int rowCountPerChunk,
+ int columnSizeInBytes, PinotDataBufferMemoryManager memoryManager, String context) {
_memoryManager = memoryManager;
_context = context;
int initialCapacity = Math.max(maxNumberOfMultiValuesPerRow, rowCountPerChunk * avgMultiValueCount);
@@ -169,23 +171,6 @@ public class FixedByteSingleColumnMultiValueReaderWriter extends BaseSingleColum
}
}
- @Override
- public void close()
- throws IOException {
- for (FixedByteSingleValueMultiColWriter writer : _headerWriters) {
- writer.close();
- }
- for (FixedByteSingleValueMultiColReader reader : _headerReaders) {
- reader.close();
- }
- for (FixedByteSingleValueMultiColWriter writer : _dataWriters) {
- writer.close();
- }
- for (FixedByteSingleValueMultiColReader reader : _dataReaders) {
- reader.close();
- }
- }
-
private void writeIntoHeader(int row, int dataWriterIndex, int startIndex, int length) {
if (row >= _headerWriters.size() * _rowCountPerChunk) {
addHeaderBuffer();
@@ -220,180 +205,146 @@ public class FixedByteSingleColumnMultiValueReaderWriter extends BaseSingleColum
return newStartIndex;
}
+ /**
+ * TODO: Currently we only support dictionary-encoded forward index on multi-value columns.
+ */
@Override
- public void setCharArray(int row, char[] charArray) {
- int newStartIndex = updateHeader(row, charArray.length);
- for (int i = 0; i < charArray.length; i++) {
- _currentDataWriter.setChar(newStartIndex + i, 0, charArray[i]);
- }
- }
-
- @Override
- public void setShortArray(int row, short[] shortsArray) {
-
- int newStartIndex = updateHeader(row, shortsArray.length);
- for (int i = 0; i < shortsArray.length; i++) {
- _currentDataWriter.setShort(newStartIndex + i, 0, shortsArray[i]);
- }
+ public boolean isDictionaryEncoded() {
+ return true;
}
@Override
- public void setIntArray(int row, int[] intArray) {
- int newStartIndex = updateHeader(row, intArray.length);
- for (int i = 0; i < intArray.length; i++) {
- _currentDataWriter.setInt(newStartIndex + i, 0, intArray[i]);
- }
+ public boolean isSingleValue() {
+ return false;
}
@Override
- public void setLongArray(int row, long[] longArray) {
- int newStartIndex = updateHeader(row, longArray.length);
- for (int i = 0; i < longArray.length; i++) {
- _currentDataWriter.setLong(newStartIndex + i, 0, longArray[i]);
- }
+ public DataType getValueType() {
+ return DataType.INT;
}
@Override
- public void setFloatArray(int row, float[] floatArray) {
- int newStartIndex = updateHeader(row, floatArray.length);
- for (int i = 0; i < floatArray.length; i++) {
- _currentDataWriter.setFloat(newStartIndex + i, 0, floatArray[i]);
- }
+ public int getLengthOfShortestElement() {
+ return _columnSizeInBytes;
}
@Override
- public void setDoubleArray(int row, double[] doubleArray) {
- int newStartIndex = updateHeader(row, doubleArray.length);
- for (int i = 0; i < doubleArray.length; i++) {
- _currentDataWriter.setDouble(newStartIndex + i, 0, doubleArray[i]);
- }
+ public int getLengthOfLongestElement() {
+ return _columnSizeInBytes;
}
@Override
- public void setStringArray(int row, String[] stringArray) {
- int newStartIndex = updateHeader(row, stringArray.length);
- for (int i = 0; i < stringArray.length; i++) {
- _currentDataWriter.setString(newStartIndex + i, 0, stringArray[i]);
+ public int getDictIdMV(int docId, int[] dictIdBuffer) {
+ FixedByteSingleValueMultiColReader headerReader = getCurrentReader(docId);
+ int rowInCurrentHeader = getRowInCurrentHeader(docId);
+ int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
+ int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
+ int length = headerReader.getInt(rowInCurrentHeader, 2);
+ FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
+ for (int i = 0; i < length; i++) {
+ dictIdBuffer[i] = dataReader.getInt(startIndex + i, 0);
}
+ return length;
}
@Override
- public void setBytesArray(int row, byte[][] bytesArray) {
- int newStartIndex = updateHeader(row, bytesArray.length);
- for (int i = 0; i < bytesArray.length; i++) {
- _currentDataWriter.setBytes(newStartIndex + i, 0, bytesArray[i]);
- }
+ public int getIntMV(int docId, int[] valueBuffer) {
+ return getDictIdMV(docId, valueBuffer);
}
@Override
- public int getCharArray(int row, char[] charArray) {
- FixedByteSingleValueMultiColReader headerReader = getCurrentReader(row);
- int rowInCurrentHeader = getRowInCurrentHeader(row);
+ public int getLongMV(int docId, long[] valueBuffer) {
+ FixedByteSingleValueMultiColReader headerReader = getCurrentReader(docId);
+ int rowInCurrentHeader = getRowInCurrentHeader(docId);
int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
int length = headerReader.getInt(rowInCurrentHeader, 2);
FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
for (int i = 0; i < length; i++) {
- charArray[i] = dataReader.getChar(startIndex + i, 0);
+ valueBuffer[i] = dataReader.getLong(startIndex + i, 0);
}
return length;
}
@Override
- public int getShortArray(int row, short[] shortsArray) {
- FixedByteSingleValueMultiColReader headerReader = getCurrentReader(row);
- int rowInCurrentHeader = getRowInCurrentHeader(row);
+ public int getFloatMV(int docId, float[] valueBuffer) {
+ FixedByteSingleValueMultiColReader headerReader = getCurrentReader(docId);
+ int rowInCurrentHeader = getRowInCurrentHeader(docId);
int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
int length = headerReader.getInt(rowInCurrentHeader, 2);
FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
for (int i = 0; i < length; i++) {
- shortsArray[i] = dataReader.getShort(startIndex + i, 0);
+ valueBuffer[i] = dataReader.getFloat(startIndex + i, 0);
}
return length;
}
@Override
- public int getIntArray(int row, int[] intArray) {
- FixedByteSingleValueMultiColReader headerReader = getCurrentReader(row);
- int rowInCurrentHeader = getRowInCurrentHeader(row);
+ public int getDoubleMV(int docId, double[] valueBuffer) {
+ FixedByteSingleValueMultiColReader headerReader = getCurrentReader(docId);
+ int rowInCurrentHeader = getRowInCurrentHeader(docId);
int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
int length = headerReader.getInt(rowInCurrentHeader, 2);
FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
for (int i = 0; i < length; i++) {
- intArray[i] = dataReader.getInt(startIndex + i, 0);
+ valueBuffer[i] = dataReader.getDouble(startIndex + i, 0);
}
return length;
}
@Override
- public int getLongArray(int row, long[] longArray) {
- FixedByteSingleValueMultiColReader headerReader = getCurrentReader(row);
- int rowInCurrentHeader = getRowInCurrentHeader(row);
- int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
- int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
- int length = headerReader.getInt(rowInCurrentHeader, 2);
- FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
- for (int i = 0; i < length; i++) {
- longArray[i] = dataReader.getLong(startIndex + i, 0);
+ public void setDictIdMV(int docId, int[] dictIds) {
+ int newStartIndex = updateHeader(docId, dictIds.length);
+ for (int i = 0; i < dictIds.length; i++) {
+ _currentDataWriter.setInt(newStartIndex + i, 0, dictIds[i]);
}
- return length;
}
@Override
- public int getFloatArray(int row, float[] floatArray) {
- FixedByteSingleValueMultiColReader headerReader = getCurrentReader(row);
- int rowInCurrentHeader = getRowInCurrentHeader(row);
- int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
- int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
- int length = headerReader.getInt(rowInCurrentHeader, 2);
- FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
- for (int i = 0; i < length; i++) {
- floatArray[i] = dataReader.getFloat(startIndex + i, 0);
+ public void setIntMV(int docId, int[] values) {
+ setDictIdMV(docId, values);
+ }
+
+ @Override
+ public void setLongMV(int docId, long[] values) {
+ int newStartIndex = updateHeader(docId, values.length);
+ for (int i = 0; i < values.length; i++) {
+ _currentDataWriter.setLong(newStartIndex + i, 0, values[i]);
}
- return length;
}
@Override
- public int getDoubleArray(int row, double[] doubleArray) {
- FixedByteSingleValueMultiColReader headerReader = getCurrentReader(row);
- int rowInCurrentHeader = getRowInCurrentHeader(row);
- int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
- int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
- int length = headerReader.getInt(rowInCurrentHeader, 2);
- FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
- for (int i = 0; i < length; i++) {
- doubleArray[i] = dataReader.getDouble(startIndex + i, 0);
+ public void setFloatMV(int docId, float[] values) {
+ int newStartIndex = updateHeader(docId, values.length);
+ for (int i = 0; i < values.length; i++) {
+ _currentDataWriter.setFloat(newStartIndex + i, 0, values[i]);
}
- return length;
}
@Override
- public int getStringArray(int row, String[] stringArray) {
- FixedByteSingleValueMultiColReader headerReader = getCurrentReader(row);
- int rowInCurrentHeader = getRowInCurrentHeader(row);
- int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
- int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
- int length = headerReader.getInt(rowInCurrentHeader, 2);
- FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
- for (int i = 0; i < length; i++) {
- stringArray[i] = dataReader.getString(startIndex + i, 0);
+ public void setDoubleMV(int docId, double[] values) {
+ int newStartIndex = updateHeader(docId, values.length);
+ for (int i = 0; i < values.length; i++) {
+ _currentDataWriter.setDouble(newStartIndex + i, 0, values[i]);
}
- return length;
}
@Override
- public int getBytesArray(int row, byte[][] bytesArray) {
- FixedByteSingleValueMultiColReader headerReader = getCurrentReader(row);
- int rowInCurrentHeader = getRowInCurrentHeader(row);
- int bufferIndex = headerReader.getInt(rowInCurrentHeader, 0);
- int startIndex = headerReader.getInt(rowInCurrentHeader, 1);
- int length = headerReader.getInt(rowInCurrentHeader, 2);
- FixedByteSingleValueMultiColReader dataReader = _dataReaders.get(bufferIndex);
- for (int i = 0; i < length; i++) {
- bytesArray[i] = dataReader.getBytes(startIndex + i, 0);
+ public void close()
+ throws IOException {
+ for (FixedByteSingleValueMultiColWriter writer : _headerWriters) {
+ writer.close();
+ }
+ for (FixedByteSingleValueMultiColReader reader : _headerReaders) {
+ reader.close();
+ }
+ for (FixedByteSingleValueMultiColWriter writer : _dataWriters) {
+ writer.close();
+ }
+ for (FixedByteSingleValueMultiColReader reader : _dataReaders) {
+ reader.close();
}
- return length;
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleColumnSingleValueReaderWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/FixedByteSVMutableForwardIndex.java
similarity index 68%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleColumnSingleValueReaderWriter.java
rename to pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/FixedByteSVMutableForwardIndex.java
index 0d3103d..775d8ab 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/FixedByteSingleColumnSingleValueReaderWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/FixedByteSVMutableForwardIndex.java
@@ -16,17 +16,18 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.readerwriter.impl;
+package org.apache.pinot.core.realtime.impl.forward;
import java.io.Closeable;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.pinot.core.io.reader.impl.FixedByteSingleValueMultiColReader;
-import org.apache.pinot.core.io.readerwriter.BaseSingleColumnSingleValueReaderWriter;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
import org.apache.pinot.core.io.writer.impl.FixedByteSingleValueMultiColWriter;
+import org.apache.pinot.core.segment.index.readers.MutableForwardIndex;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -40,135 +41,165 @@ import org.slf4j.LoggerFactory;
* are not guaranteed to have a deterministic value. </li>
* </ul>
*/
-// TODO: Check thread-safety
-public class FixedByteSingleColumnSingleValueReaderWriter extends BaseSingleColumnSingleValueReaderWriter {
- private static final Logger LOGGER = LoggerFactory.getLogger(FixedByteSingleColumnSingleValueReaderWriter.class);
+// TODO: Fix thread-safety issue for ArrayList
+// TODO: Optimize it
+public class FixedByteSVMutableForwardIndex implements MutableForwardIndex {
+ private static final Logger LOGGER = LoggerFactory.getLogger(FixedByteSVMutableForwardIndex.class);
private final List<WriterWithOffset> _writers = new ArrayList<>();
private final List<ReaderWithOffset> _readers = new ArrayList<>();
- private final long _chunkSizeInBytes;
+ private final boolean _dictionaryEncoded;
+ private final DataType _valueType;
+ private final int _valueSizeInBytes;
private final int _numRowsPerChunk;
- private final int _columnSizesInBytes;
+ private final long _chunkSizeInBytes;
private final PinotDataBufferMemoryManager _memoryManager;
private final String _allocationContext;
private int _capacityInRows = 0;
/**
+ * @param valueType Data type of the values
* @param numRowsPerChunk Number of rows to pack in one chunk before a new chunk is created.
- * @param columnSizesInBytes Size of column value in bytes.
* @param memoryManager Memory manager to be used for allocating memory.
* @param allocationContext Allocation allocationContext.
*/
- public FixedByteSingleColumnSingleValueReaderWriter(int numRowsPerChunk, int columnSizesInBytes,
+ public FixedByteSVMutableForwardIndex(boolean dictionaryEncoded, DataType valueType, int numRowsPerChunk,
PinotDataBufferMemoryManager memoryManager, String allocationContext) {
- _chunkSizeInBytes = numRowsPerChunk * columnSizesInBytes;
+ _dictionaryEncoded = dictionaryEncoded;
+ _valueType = valueType;
+ _valueSizeInBytes = valueType.size();
_numRowsPerChunk = numRowsPerChunk;
- _columnSizesInBytes = columnSizesInBytes;
+ _chunkSizeInBytes = numRowsPerChunk * _valueSizeInBytes;
_memoryManager = memoryManager;
_allocationContext = allocationContext;
addBuffer();
}
@Override
+ public boolean isDictionaryEncoded() {
+ return _dictionaryEncoded;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return true;
+ }
+
+ @Override
+ public DataType getValueType() {
+ return _valueType;
+ }
+
+ @Override
public int getLengthOfShortestElement() {
- return _columnSizesInBytes;
+ return _valueSizeInBytes;
}
@Override
public int getLengthOfLongestElement() {
- return _columnSizesInBytes;
+ return _valueSizeInBytes;
}
@Override
- public void close()
- throws IOException {
- for (WriterWithOffset writer : _writers) {
- writer.close();
- }
- for (ReaderWithOffset reader : _readers) {
- reader.close();
+ public int getDictId(int docId) {
+ int bufferId = getBufferId(docId);
+ return _readers.get(bufferId).getInt(docId);
+ }
+
+ @Override
+ public void readDictIds(int[] docIds, int length, int[] dictIdBuffer) {
+ /*
+ * TODO
+ * If we assume that the document ids are sorted, then we can write logic to move values from one reader
+ * at a time, identifying the rows in sequence that belong to the same block. This logic is more complex, but may
+ * perform better in the sorted case.
+ *
+ * An alternative is to not have multiple _dataBuffers, but just copy the values from one buffer to next as we
+ * increase the number of rows.
+ */
+ if (_readers.size() == 1) {
+ _readers.get(0).getReader().readIntValues(docIds, 0, 0, length, dictIdBuffer, 0);
+ } else {
+ for (int i = 0; i < length; i++) {
+ int docId = docIds[i];
+ dictIdBuffer[i] = _readers.get(getBufferId(docId)).getInt(docId);
+ }
}
}
@Override
- public void setInt(int row, int i) {
- addBufferIfNeeded(row);
- getWriterForRow(row).setInt(row, i);
+ public int getInt(int docId) {
+ int bufferId = getBufferId(docId);
+ return _readers.get(bufferId).getInt(docId);
}
@Override
- public void setLong(int row, long l) {
- addBufferIfNeeded(row);
- getWriterForRow(row).setLong(row, l);
+ public long getLong(int docId) {
+ int bufferId = getBufferId(docId);
+ return _readers.get(bufferId).getLong(docId);
}
@Override
- public void setFloat(int row, float f) {
- addBufferIfNeeded(row);
- getWriterForRow(row).setFloat(row, f);
+ public float getFloat(int docId) {
+ int bufferId = getBufferId(docId);
+ return _readers.get(bufferId).getFloat(docId);
}
@Override
- public void setDouble(int row, double d) {
- addBufferIfNeeded(row);
- getWriterForRow(row).setDouble(row, d);
+ public double getDouble(int docId) {
+ int bufferId = getBufferId(docId);
+ return _readers.get(bufferId).getDouble(docId);
}
- private WriterWithOffset getWriterForRow(int row) {
- return _writers.get(getBufferId(row));
+ private int getBufferId(int row) {
+ return row / _numRowsPerChunk;
}
@Override
- public int getInt(int row) {
- int bufferId = getBufferId(row);
- return _readers.get(bufferId).getInt(row);
+ public void setDictId(int docId, int dictId) {
+ addBufferIfNeeded(docId);
+ getWriterForRow(docId).setInt(docId, dictId);
}
@Override
- public long getLong(int row) {
- int bufferId = getBufferId(row);
- return _readers.get(bufferId).getLong(row);
+ public void setInt(int docId, int value) {
+ addBufferIfNeeded(docId);
+ getWriterForRow(docId).setInt(docId, value);
}
@Override
- public float getFloat(int row) {
- int bufferId = getBufferId(row);
- return _readers.get(bufferId).getFloat(row);
+ public void setLong(int docId, long value) {
+ addBufferIfNeeded(docId);
+ getWriterForRow(docId).setLong(docId, value);
}
@Override
- public double getDouble(int row) {
- int bufferId = getBufferId(row);
- return _readers.get(bufferId).getDouble(row);
+ public void setFloat(int docId, float value) {
+ addBufferIfNeeded(docId);
+ getWriterForRow(docId).setFloat(docId, value);
}
@Override
- public void readValues(int[] rows, int rowStartPos, int rowSize, int[] values, int valuesStartPos) {
- /*
- * TODO
- * If we assume that the row IDs in 'rows' is sorted, then we can write logic to move values from one reader
- * at a time, identifying the rows in sequence that belong to the same block. This logic is more complex, but may
- * perform better in the sorted case.
- *
- * An alternative is to not have multiple _dataBuffers, but just copy the values from one buffer to next as we
- * increase the number of rows.
- */
- if (_readers.size() == 1) {
- _readers.get(0).getReader().readIntValues(rows, 0, rowStartPos, rowSize, values, valuesStartPos);
- } else {
- for (int rowIter = rowStartPos, valueIter = valuesStartPos; rowIter < rowStartPos + rowSize;
- rowIter++, valueIter++) {
- int row = rows[rowIter];
- int bufferId = getBufferId(row);
- values[valueIter] = _readers.get(bufferId).getInt(row);
- }
- }
+ public void setDouble(int docId, double value) {
+ addBufferIfNeeded(docId);
+ getWriterForRow(docId).setDouble(docId, value);
}
- private int getBufferId(int row) {
- return row / _numRowsPerChunk;
+ private WriterWithOffset getWriterForRow(int row) {
+ return _writers.get(getBufferId(row));
+ }
+
+ @Override
+ public void close()
+ throws IOException {
+ for (WriterWithOffset writer : _writers) {
+ writer.close();
+ }
+ for (ReaderWithOffset reader : _readers) {
+ reader.close();
+ }
}
private void addBuffer() {
@@ -176,10 +207,10 @@ public class FixedByteSingleColumnSingleValueReaderWriter extends BaseSingleColu
// NOTE: PinotDataBuffer is tracked in the PinotDataBufferMemoryManager. No need to track it inside the class.
PinotDataBuffer buffer = _memoryManager.allocate(_chunkSizeInBytes, _allocationContext);
_writers.add(
- new WriterWithOffset(new FixedByteSingleValueMultiColWriter(buffer, /*cols=*/1, new int[]{_columnSizesInBytes}),
+ new WriterWithOffset(new FixedByteSingleValueMultiColWriter(buffer, /*cols=*/1, new int[]{_valueSizeInBytes}),
_capacityInRows));
_readers.add(new ReaderWithOffset(
- new FixedByteSingleValueMultiColReader(buffer, _numRowsPerChunk, new int[]{_columnSizesInBytes}),
+ new FixedByteSingleValueMultiColReader(buffer, _numRowsPerChunk, new int[]{_valueSizeInBytes}),
_capacityInRows));
_capacityInRows += _numRowsPerChunk;
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/VarByteSingleColumnSingleValueReaderWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/VarByteSVMutableForwardIndex.java
similarity index 53%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/VarByteSingleColumnSingleValueReaderWriter.java
rename to pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/VarByteSVMutableForwardIndex.java
index 0327b12..bd03ade 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/readerwriter/impl/VarByteSingleColumnSingleValueReaderWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/forward/VarByteSVMutableForwardIndex.java
@@ -16,105 +16,84 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.readerwriter.impl;
+package org.apache.pinot.core.realtime.impl.forward;
import java.io.IOException;
import org.apache.pinot.common.utils.StringUtil;
-import org.apache.pinot.core.io.readerwriter.BaseSingleColumnSingleValueReaderWriter;
import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;
import org.apache.pinot.core.io.writer.impl.MutableOffHeapByteArrayStore;
+import org.apache.pinot.core.segment.index.readers.MutableForwardIndex;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
-public class VarByteSingleColumnSingleValueReaderWriter extends BaseSingleColumnSingleValueReaderWriter {
+
+/**
+ * Single-value forward index reader-writer for variable length values (STRING and BYTES).
+ */
+public class VarByteSVMutableForwardIndex implements MutableForwardIndex {
+ private final DataType _valueType;
private final MutableOffHeapByteArrayStore _byteArrayStore;
private int _lengthOfShortestElement;
private int _lengthOfLongestElement;
- public VarByteSingleColumnSingleValueReaderWriter(
- PinotDataBufferMemoryManager memoryManager,
- String allocationContext,
- int estimatedMaxNumberOfValues,
- int estimatedAverageStringLength) {
- _byteArrayStore = new MutableOffHeapByteArrayStore(memoryManager, allocationContext, estimatedMaxNumberOfValues, estimatedAverageStringLength);
+ public VarByteSVMutableForwardIndex(DataType valueType, PinotDataBufferMemoryManager memoryManager,
+ String allocationContext, int estimatedMaxNumberOfValues, int estimatedAverageStringLength) {
+ _valueType = valueType;
+ _byteArrayStore = new MutableOffHeapByteArrayStore(memoryManager, allocationContext, estimatedMaxNumberOfValues,
+ estimatedAverageStringLength);
_lengthOfShortestElement = Integer.MAX_VALUE;
_lengthOfLongestElement = Integer.MIN_VALUE;
}
@Override
- public int getLengthOfShortestElement() {
- return _lengthOfShortestElement;
- }
-
- @Override
- public int getLengthOfLongestElement() {
- return _lengthOfLongestElement;
- }
-
- @Override
- public void close()
- throws IOException {
- _byteArrayStore.close();
- }
-
- @Override
- public void setInt(int row, int i) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public int getInt(int row) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void setLong(int row, long l) {
- throw new UnsupportedOperationException();
+ public boolean isDictionaryEncoded() {
+ return false;
}
@Override
- public long getLong(int row) {
- throw new UnsupportedOperationException();
+ public boolean isSingleValue() {
+ return true;
}
@Override
- public void setFloat(int row, float f) {
- throw new UnsupportedOperationException();
+ public DataType getValueType() {
+ return _valueType;
}
@Override
- public float getFloat(int row) {
- throw new UnsupportedOperationException();
+ public int getLengthOfShortestElement() {
+ return _lengthOfShortestElement;
}
@Override
- public void setDouble(int row, double d) {
- throw new UnsupportedOperationException();
+ public int getLengthOfLongestElement() {
+ return _lengthOfLongestElement;
}
@Override
- public double getDouble(int row) {
- throw new UnsupportedOperationException();
+ public String getString(int docId) {
+ return StringUtil.decodeUtf8(_byteArrayStore.get(docId));
}
@Override
- public void setString(int row, String val) {
- byte[] serializedValue = StringUtil.encodeUtf8(val);
- setBytes(row, serializedValue);
+ public byte[] getBytes(int docId) {
+ return _byteArrayStore.get(docId);
}
@Override
- public String getString(int row) {
- return StringUtil.decodeUtf8(_byteArrayStore.get(row));
+ public void setString(int docId, String value) {
+ setBytes(docId, StringUtil.encodeUtf8(value));
}
@Override
- public void setBytes(int row, byte[] value) {
+ public void setBytes(int docId, byte[] value) {
_byteArrayStore.add(value);
_lengthOfLongestElement = Math.max(_lengthOfLongestElement, value.length);
_lengthOfShortestElement = Math.min(_lengthOfShortestElement, value.length);
}
@Override
- public byte[] getBytes(int row) {
- return _byteArrayStore.get(row);
+ public void close()
+ throws IOException {
+ _byteArrayStore.close();
}
-}
\ No newline at end of file
+}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java
index ec6b5c4..e66ba76 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/invertedindex/RealtimeInvertedIndexReader.java
@@ -61,13 +61,6 @@ public class RealtimeInvertedIndexReader implements InvertedIndexReader<MutableR
}
@Override
- public MutableRoaringBitmap getDocIds(Object value) {
- // This should not be called from anywhere. If it happens, there is a bug
- // and that's why we throw illegal state exception
- throw new IllegalStateException("realtime bitmap inverted index reader supports lookup only on dictionary id");
- }
-
- @Override
public MutableRoaringBitmap getDocIds(int dictId) {
ThreadSafeMutableRoaringBitmap bitmap;
try {
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/invertedindex/RealtimeLuceneTextIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/invertedindex/RealtimeLuceneTextIndexReader.java
index 3d029c7..3099fc3 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/invertedindex/RealtimeLuceneTextIndexReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/invertedindex/RealtimeLuceneTextIndexReader.java
@@ -29,7 +29,6 @@ import org.apache.lucene.search.Query;
import org.apache.lucene.search.SearcherManager;
import org.apache.pinot.core.segment.creator.impl.inv.text.LuceneTextIndexCreator;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
-import org.apache.pinot.core.segment.index.readers.text.LuceneDocIdCollector;
import org.roaringbitmap.IntIterator;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
import org.slf4j.LoggerFactory;
@@ -90,14 +89,8 @@ public class RealtimeLuceneTextIndexReader implements InvertedIndexReader<Mutabl
throw new UnsupportedOperationException("Using dictionary ID is not supported on Lucene inverted index");
}
- /**
- * Get docIds from the text inverted index for a given raw value
- * @param value value to look for in the inverted index
- * @return docIDs in bitmap
- */
@Override
- public MutableRoaringBitmap getDocIds(Object value) {
- String searchQuery = (String) value;
+ public MutableRoaringBitmap getDocIds(String searchQuery) {
MutableRoaringBitmap docIDs = new MutableRoaringBitmap();
Collector docIDCollector = new RealtimeLuceneDocIdCollector(docIDs);
IndexSearcher indexSearcher = null;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/BaseSingleValueRawIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/BaseSingleValueRawIndexCreator.java
deleted file mode 100644
index 90e9a4e..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/BaseSingleValueRawIndexCreator.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.segment.creator;
-
-/**
- * Abstract base class implementation for {@link SingleValueRawIndexCreator}
- */
-public abstract class BaseSingleValueRawIndexCreator implements SingleValueRawIndexCreator {
- @Override
- public void index(int docId, int valueToIndex) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void index(int docId, long valueToIndex) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void index(int docId, float valueToIndex) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void index(int docId, double valueToIndex) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void index(int docId, String valueToIndex) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void index(int docId, byte[] valueToIndex) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void index(int docId, Object valueToIndex) {
- throw new UnsupportedOperationException();
- }
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/ForwardIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/ForwardIndexCreator.java
index 192d573..62e765c 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/ForwardIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/ForwardIndexCreator.java
@@ -19,7 +19,158 @@
package org.apache.pinot.core.segment.creator;
import java.io.Closeable;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+/**
+ * Interface for forward index creator.
+ */
public interface ForwardIndexCreator extends Closeable {
+
+ /**
+ * Returns {@code true} if the forward index is dictionary-encoded, {@code false} if it is raw.
+ */
+ boolean isDictionaryEncoded();
+
+ /**
+ * Returns {@code true} if the forward index is for a single-value column, {@code false} if it is for a multi-value
+ * column.
+ */
+ boolean isSingleValue();
+
+ /**
+ * Returns the data type of the values in the forward index. Returns {@link DataType#INT} for dictionary-encoded
+ * forward index.
+ */
+ DataType getValueType();
+
+ /**
+ * DICTIONARY-ENCODED INDEX APIs
+ */
+
+ /**
+ * Writes the dictionary id for the next single-value into the forward index.
+ *
+ * @param dictId Document id to write
+ */
+ default void putDictId(int dictId) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the dictionary ids for the next multi-value into the forward index.
+ *
+ * @param dictIds Document ids to write
+ */
+ default void putDictIdMV(int[] dictIds) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * SINGLE-VALUE COLUMN RAW INDEX APIs
+ */
+
+ /**
+ * Writes the next INT type single-value into the forward index.
+ *
+ * @param value Value to write
+ */
+ default void putInt(int value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next LONG type single-value into the forward index.
+ *
+ * @param value Value to write
+ */
+ default void putLong(long value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next FLOAT type single-value into the forward index.
+ *
+ * @param value Value to write
+ */
+ default void putFloat(float value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next DOUBLE type single-value into the forward index.
+ *
+ * @param value Value to write
+ */
+ default void putDouble(double value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next STRING type single-value into the forward index.
+ *
+ * @param value Value to write
+ */
+ default void putString(String value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next BYTES type single-value into the forward index.
+ *
+ * @param value Value to write
+ */
+ default void putBytes(byte[] value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * MULTI-VALUE COLUMN RAW INDEX APIs
+ * TODO: Not supported yet
+ */
+
+ /**
+ * Writes the next INT type multi-value into the forward index.
+ *
+ * @param values Values to write
+ */
+ default void putIntMV(int[] values) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next LONG type multi-value into the forward index.
+ *
+ * @param values Values to write
+ */
+ default void putLongMV(long[] values) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next FLOAT type multi-value into the forward index.
+ *
+ * @param values Values to write
+ */
+ default void putFloatMV(float[] values) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next DOUBLE type multi-value into the forward index.
+ *
+ * @param values Values to write
+ */
+ default void putDoubleMV(double[] values) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the next STRING type multi-value into the forward index.
+ *
+ * @param values Values to write
+ */
+ default void putStringMV(String[] values) {
+ throw new UnsupportedOperationException();
+ }
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/MultiValueForwardIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/MultiValueForwardIndexCreator.java
deleted file mode 100644
index 3008159..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/MultiValueForwardIndexCreator.java
+++ /dev/null
@@ -1,30 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.segment.creator;
-
-public interface MultiValueForwardIndexCreator extends ForwardIndexCreator {
-
- /**
- * For multi-valued column, adds the dictionary Ids to the document.
- *
- * @param docId Document Id
- * @param dictIds Array of dictionary Ids
- */
- void index(int docId, int[] dictIds);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/SingleValueForwardIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/SingleValueForwardIndexCreator.java
deleted file mode 100644
index 7eb0cef..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/SingleValueForwardIndexCreator.java
+++ /dev/null
@@ -1,30 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.segment.creator;
-
-public interface SingleValueForwardIndexCreator extends ForwardIndexCreator {
-
- /**
- * For single-valued column, adds the dictionary Id to the document.
- *
- * @param docId Document Id
- * @param dictId Dictionary Id
- */
- void index(int docId, int dictId);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/SingleValueRawIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/SingleValueRawIndexCreator.java
deleted file mode 100644
index c1184bf..0000000
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/SingleValueRawIndexCreator.java
+++ /dev/null
@@ -1,84 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.pinot.core.segment.creator;
-
-/**
- * Interface for index creator for single-values with variable length (eg strings).
- * Implementations of this interface write raw data values (in potentially compressed formats),
- * as opposed to dictionary encoded index.
- */
-public interface SingleValueRawIndexCreator extends ForwardIndexCreator {
-
- /**
- * This method creates an index for the given docId and int value.
- *
- * @param docId Document id
- * @param valueToIndex int value to index.
- */
- void index(int docId, int valueToIndex);
-
- /**
- * This method creates an index for the given docId and long value.
- *
- * @param docId Document id
- * @param valueToIndex long value to index.
- */
- void index(int docId, long valueToIndex);
-
- /**
- * This method creates an index for the given docId and float value.
- *
- * @param docId Document id
- * @param valueToIndex float value to index.
- */
- void index(int docId, float valueToIndex);
-
- /**
- * This method creates an index for the given docId and double value.
- *
- * @param docId Document id
- * @param valueToIndex double value to index.
- */
- void index(int docId, double valueToIndex);
-
- /**
- * This method creates an index for the given docId and String value.
- *
- * @param docId Document id
- * @param valueToIndex Variable length string value to index.
- */
- void index(int docId, String valueToIndex);
-
- /**
- * This method creates an index for the given docId and byte[] value.
- *
- * @param docId Document id
- * @param valueToIndex Variable length byte[] value to index.
- */
- void index(int docId, byte[] valueToIndex);
-
- /**
- * This method creates an index for the given docId and object value.
- * Object should be one of the primitive supported types.
- *
- * @param docId Document id
- * @param valueToIndex Object value to index.
- */
- void index(int docId, Object valueToIndex);
-}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
index 25c20c7..06146f3 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
@@ -35,15 +35,12 @@ import org.apache.pinot.core.data.partition.PartitionFunction;
import org.apache.pinot.core.indexsegment.generator.SegmentGeneratorConfig;
import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
import org.apache.pinot.core.io.util.PinotDataBitSet;
-import org.apache.pinot.core.io.writer.impl.v1.BaseChunkSingleValueWriter;
+import org.apache.pinot.core.io.writer.impl.BaseChunkSVForwardIndexWriter;
import org.apache.pinot.core.segment.creator.ColumnIndexCreationInfo;
import org.apache.pinot.core.segment.creator.ForwardIndexCreator;
import org.apache.pinot.core.segment.creator.InvertedIndexCreator;
-import org.apache.pinot.core.segment.creator.MultiValueForwardIndexCreator;
import org.apache.pinot.core.segment.creator.SegmentCreator;
import org.apache.pinot.core.segment.creator.SegmentIndexCreationInfo;
-import org.apache.pinot.core.segment.creator.SingleValueForwardIndexCreator;
-import org.apache.pinot.core.segment.creator.SingleValueRawIndexCreator;
import org.apache.pinot.core.segment.creator.TextIndexType;
import org.apache.pinot.core.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
import org.apache.pinot.core.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
@@ -57,6 +54,7 @@ import org.apache.pinot.core.segment.creator.impl.nullvalue.NullValueVectorCreat
import org.apache.pinot.spi.config.table.FieldConfig;
import org.apache.pinot.spi.data.DateTimeFieldSpec;
import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.apache.pinot.spi.data.FieldSpec.FieldType;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.data.readers.GenericRow;
@@ -195,7 +193,8 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
getColumnCompressionType(segmentCreationSpec, fieldSpec);
// Initialize forward index creator
- boolean deriveNumDocsPerChunk = shouldDeriveNumDocsPerChunk(columnName, segmentCreationSpec.getColumnProperties());
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(columnName, segmentCreationSpec.getColumnProperties());
int writerVersion = rawIndexWriterVersion(columnName, segmentCreationSpec.getColumnProperties());
_forwardIndexCreatorMap.put(columnName,
getRawIndexCreatorForColumn(_indexDir, compressionType, columnName, fieldSpec.getDataType(), totalDocs,
@@ -217,10 +216,12 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
}
}
- public static boolean shouldDeriveNumDocsPerChunk(String columnName, Map<String, Map<String, String>> columnProperties) {
+ public static boolean shouldDeriveNumDocsPerChunk(String columnName,
+ Map<String, Map<String, String>> columnProperties) {
if (columnProperties != null) {
Map<String, String> properties = columnProperties.get(columnName);
- return properties != null && Boolean.parseBoolean(properties.get(FieldConfig.DERIVE_NUM_DOCS_PER_CHUNK_RAW_INDEX_KEY));
+ return properties != null && Boolean
+ .parseBoolean(properties.get(FieldConfig.DERIVE_NUM_DOCS_PER_CHUNK_RAW_INDEX_KEY));
}
return false;
}
@@ -230,11 +231,11 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
Map<String, String> properties = columnProperties.get(columnName);
String version = properties.get(FieldConfig.RAW_INDEX_WRITER_VERSION);
if (version == null) {
- return BaseChunkSingleValueWriter.DEFAULT_VERSION;
+ return BaseChunkSVForwardIndexWriter.DEFAULT_VERSION;
}
return Integer.parseInt(version);
}
- return BaseChunkSingleValueWriter.DEFAULT_VERSION;
+ return BaseChunkSVForwardIndexWriter.DEFAULT_VERSION;
}
/**
@@ -296,7 +297,7 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
"Creation of indices without dictionaries is supported for single valued columns only.");
}
return false;
- } else if (spec.getDataType().equals(FieldSpec.DataType.BYTES) && !info.isFixedLength()) {
+ } else if (spec.getDataType().equals(DataType.BYTES) && !info.isFixedLength()) {
return false;
}
return info.isCreateDictionary();
@@ -304,7 +305,10 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
@Override
public void indexRow(GenericRow row) {
- for (String columnName : _forwardIndexCreatorMap.keySet()) {
+ for (Map.Entry<String, ForwardIndexCreator> entry : _forwardIndexCreatorMap.entrySet()) {
+ String columnName = entry.getKey();
+ ForwardIndexCreator forwardIndexCreator = entry.getValue();
+
Object columnValueToIndex = row.getValue(columnName);
if (columnValueToIndex == null) {
throw new RuntimeException("Null value for column:" + columnName);
@@ -320,17 +324,38 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
// get dictID from dictionary
int dictId = dictionaryCreator.indexOfSV(columnValueToIndex);
// store the docID -> dictID mapping in forward index
- ((SingleValueForwardIndexCreator) _forwardIndexCreatorMap.get(columnName)).index(docIdCounter, dictId);
- if (_invertedIndexCreatorMap.containsKey(columnName)) {
+ forwardIndexCreator.putDictId(dictId);
+ InvertedIndexCreator invertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
+ if (invertedIndexCreator != null) {
// if inverted index enabled during segment creation,
// then store dictID -> docID mapping in inverted index
- _invertedIndexCreatorMap.get(columnName).add(dictId);
+ invertedIndexCreator.add(dictId);
}
} else {
// non-dictionary encoded SV column
// store the docId -> raw value mapping in forward index
- ((SingleValueRawIndexCreator) _forwardIndexCreatorMap.get(columnName))
- .index(docIdCounter, columnValueToIndex);
+ switch (forwardIndexCreator.getValueType()) {
+ case INT:
+ forwardIndexCreator.putInt((int) columnValueToIndex);
+ break;
+ case LONG:
+ forwardIndexCreator.putLong((long) columnValueToIndex);
+ break;
+ case FLOAT:
+ forwardIndexCreator.putFloat((float) columnValueToIndex);
+ break;
+ case DOUBLE:
+ forwardIndexCreator.putDouble((double) columnValueToIndex);
+ break;
+ case STRING:
+ forwardIndexCreator.putString((String) columnValueToIndex);
+ break;
+ case BYTES:
+ forwardIndexCreator.putBytes((byte[]) columnValueToIndex);
+ break;
+ default:
+ throw new IllegalStateException();
+ }
// text-search enabled column
if (_textIndexColumns.contains(columnName)) {
InvertedIndexCreator textInvertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
@@ -341,9 +366,10 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
} else {
// MV column (always dictionary encoded)
int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
- ((MultiValueForwardIndexCreator) _forwardIndexCreatorMap.get(columnName)).index(docIdCounter, dictIds);
- if (_invertedIndexCreatorMap.containsKey(columnName)) {
- _invertedIndexCreatorMap.get(columnName).add(dictIds, dictIds.length);
+ forwardIndexCreator.putDictIdMV(dictIds);
+ InvertedIndexCreator invertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
+ if (invertedIndexCreator != null) {
+ invertedIndexCreator.add(dictIds, dictIds.length);
}
}
@@ -422,10 +448,12 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
}
if (!config.isSkipTimeValueCheck()) {
- Interval timeInterval = new Interval(timeUnit.toMillis(startTime), timeUnit.toMillis(endTime), DateTimeZone.UTC);
+ Interval timeInterval =
+ new Interval(timeUnit.toMillis(startTime), timeUnit.toMillis(endTime), DateTimeZone.UTC);
Preconditions.checkState(TimeUtils.isValidTimeInterval(timeInterval),
"Invalid segment start/end time: %s (in millis: %s/%s) for time column: %s, must be between: %s",
- timeInterval, timeInterval.getStartMillis(), timeInterval.getEndMillis(), timeColumnName, TimeUtils.VALID_TIME_INTERVAL);
+ timeInterval, timeInterval.getStartMillis(), timeInterval.getEndMillis(), timeColumnName,
+ TimeUtils.VALID_TIME_INTERVAL);
}
properties.setProperty(SEGMENT_START_TIME, startTime);
@@ -561,45 +589,24 @@ public class SegmentColumnarIndexCreator implements SegmentCreator {
* @return raw index creator
* @throws IOException
*/
- public static SingleValueRawIndexCreator getRawIndexCreatorForColumn(File file,
- ChunkCompressorFactory.CompressionType compressionType, String column, FieldSpec.DataType dataType, int totalDocs,
+ public static ForwardIndexCreator getRawIndexCreatorForColumn(File file,
+ ChunkCompressorFactory.CompressionType compressionType, String column, DataType dataType, int totalDocs,
int lengthOfLongestEntry, boolean deriveNumDocsPerChunk, int writerVersion)
throws IOException {
-
- SingleValueRawIndexCreator indexCreator;
switch (dataType) {
case INT:
- indexCreator = new SingleValueFixedByteRawIndexCreator(file, compressionType, column, totalDocs, Integer.BYTES,
- writerVersion);
- break;
-
case LONG:
- indexCreator = new SingleValueFixedByteRawIndexCreator(file, compressionType, column, totalDocs, Long.BYTES,
- writerVersion);
- break;
-
case FLOAT:
- indexCreator = new SingleValueFixedByteRawIndexCreator(file, compressionType, column, totalDocs, Float.BYTES,
- writerVersion);
- break;
-
case DOUBLE:
- indexCreator = new SingleValueFixedByteRawIndexCreator(file, compressionType, column, totalDocs, Double.BYTES,
+ return new SingleValueFixedByteRawIndexCreator(file, compressionType, column, totalDocs, dataType,
writerVersion);
- break;
-
case STRING:
case BYTES:
- indexCreator =
- new SingleValueVarByteRawIndexCreator(file, compressionType, column, totalDocs, lengthOfLongestEntry,
- deriveNumDocsPerChunk, writerVersion);
- break;
-
+ return new SingleValueVarByteRawIndexCreator(file, compressionType, column, totalDocs, dataType,
+ lengthOfLongestEntry, deriveNumDocsPerChunk, writerVersion);
default:
throw new UnsupportedOperationException("Data type not supported for raw indexing: " + dataType);
}
-
- return indexCreator;
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/MultiValueUnsortedForwardIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/MultiValueUnsortedForwardIndexCreator.java
index fd33bfd..6bfee8b 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/MultiValueUnsortedForwardIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/MultiValueUnsortedForwardIndexCreator.java
@@ -21,26 +21,44 @@ package org.apache.pinot.core.segment.creator.impl.fwd;
import java.io.File;
import java.io.IOException;
import org.apache.pinot.core.io.util.PinotDataBitSet;
-import org.apache.pinot.core.io.writer.SingleColumnMultiValueWriter;
-import org.apache.pinot.core.io.writer.impl.v1.FixedBitMultiValueWriter;
-import org.apache.pinot.core.segment.creator.MultiValueForwardIndexCreator;
+import org.apache.pinot.core.io.writer.impl.FixedBitMVForwardIndexWriter;
+import org.apache.pinot.core.segment.creator.ForwardIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
-public class MultiValueUnsortedForwardIndexCreator implements MultiValueForwardIndexCreator {
- private final SingleColumnMultiValueWriter _writer;
+/**
+ * Forward index creator for dictionary-encoded multi-value column.
+ */
+public class MultiValueUnsortedForwardIndexCreator implements ForwardIndexCreator {
+ private final FixedBitMVForwardIndexWriter _writer;
public MultiValueUnsortedForwardIndexCreator(File outputDir, String column, int cardinality, int numDocs,
int totalNumValues)
throws Exception {
File indexFile = new File(outputDir, column + V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION);
- _writer = new FixedBitMultiValueWriter(indexFile, numDocs, totalNumValues,
+ _writer = new FixedBitMVForwardIndexWriter(indexFile, numDocs, totalNumValues,
PinotDataBitSet.getNumBitsPerValue(cardinality - 1));
}
@Override
- public void index(int docId, int[] dictIds) {
- _writer.setIntArray(docId, dictIds);
+ public boolean isDictionaryEncoded() {
+ return true;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return false;
+ }
+
+ @Override
+ public DataType getValueType() {
+ return DataType.INT;
+ }
+
+ @Override
+ public void putDictIdMV(int[] dictIds) {
+ _writer.putDictIds(dictIds);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueFixedByteRawIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueFixedByteRawIndexCreator.java
index 7f6fc57..409b167 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueFixedByteRawIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueFixedByteRawIndexCreator.java
@@ -21,24 +21,22 @@ package org.apache.pinot.core.segment.creator.impl.fwd;
import java.io.File;
import java.io.IOException;
import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
-import org.apache.pinot.core.io.writer.impl.FixedByteSingleValueMultiColWriter;
-import org.apache.pinot.core.io.writer.impl.v1.BaseChunkSingleValueWriter;
-import org.apache.pinot.core.io.writer.impl.v1.FixedByteChunkSingleValueWriter;
-import org.apache.pinot.core.segment.creator.BaseSingleValueRawIndexCreator;
+import org.apache.pinot.core.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.core.io.writer.impl.FixedByteChunkSVForwardIndexWriter;
+import org.apache.pinot.core.segment.creator.ForwardIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
/**
- * Implementation of {@link org.apache.pinot.core.segment.creator.SingleValueRawIndexCreator}
- * <ul>
- * <li> Uses {@link FixedByteSingleValueMultiColWriter} as the underlying writer. </li>
- * <li> Should be used for writing fixed byte data (int, long, float & double). </li>
- * </ul>
+ * Forward index creator for raw (non-dictionary-encoded) single-value column of fixed length data type (INT, LONG,
+ * FLOAT, DOUBLE).
*/
-public class SingleValueFixedByteRawIndexCreator extends BaseSingleValueRawIndexCreator {
+public class SingleValueFixedByteRawIndexCreator implements ForwardIndexCreator {
private static final int NUM_DOCS_PER_CHUNK = 1000; // TODO: Auto-derive this based on metadata.
- final FixedByteChunkSingleValueWriter _indexWriter;
+ private final FixedByteChunkSVForwardIndexWriter _indexWriter;
+ private final DataType _valueType;
/**
* Constructor for the class
@@ -47,13 +45,13 @@ public class SingleValueFixedByteRawIndexCreator extends BaseSingleValueRawIndex
* @param compressionType Type of compression to use
* @param column Name of column to index
* @param totalDocs Total number of documents to index
- * @param sizeOfEntry Size of entry (in bytes)
+ * @param valueType Type of the values
* @throws IOException
*/
public SingleValueFixedByteRawIndexCreator(File baseIndexDir, ChunkCompressorFactory.CompressionType compressionType,
- String column, int totalDocs, int sizeOfEntry)
+ String column, int totalDocs, DataType valueType)
throws IOException {
- this(baseIndexDir, compressionType, column, totalDocs, sizeOfEntry, BaseChunkSingleValueWriter.DEFAULT_VERSION);
+ this(baseIndexDir, compressionType, column, totalDocs, valueType, BaseChunkSVForwardIndexWriter.DEFAULT_VERSION);
}
/**
@@ -63,52 +61,53 @@ public class SingleValueFixedByteRawIndexCreator extends BaseSingleValueRawIndex
* @param compressionType Type of compression to use
* @param column Name of column to index
* @param totalDocs Total number of documents to index
- * @param sizeOfEntry Size of entry (in bytes)
+ * @param valueType Type of the values
* @param writerVersion writer format version
* @throws IOException
*/
public SingleValueFixedByteRawIndexCreator(File baseIndexDir, ChunkCompressorFactory.CompressionType compressionType,
- String column, int totalDocs, int sizeOfEntry, int writerVersion)
+ String column, int totalDocs, DataType valueType, int writerVersion)
throws IOException {
File file = new File(baseIndexDir, column + V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION);
_indexWriter =
- new FixedByteChunkSingleValueWriter(file, compressionType, totalDocs, NUM_DOCS_PER_CHUNK, sizeOfEntry, writerVersion);
+ new FixedByteChunkSVForwardIndexWriter(file, compressionType, totalDocs, NUM_DOCS_PER_CHUNK, valueType.size(),
+ writerVersion);
+ _valueType = valueType;
}
@Override
- public void index(int docId, int valueToIndex) {
- _indexWriter.setInt(docId, valueToIndex);
+ public boolean isDictionaryEncoded() {
+ return false;
}
@Override
- public void index(int docId, long valueToIndex) {
- _indexWriter.setLong(docId, valueToIndex);
+ public boolean isSingleValue() {
+ return true;
}
@Override
- public void index(int docId, float valueToIndex) {
- _indexWriter.setFloat(docId, valueToIndex);
+ public DataType getValueType() {
+ return _valueType;
}
@Override
- public void index(int docId, double valueToIndex) {
- _indexWriter.setDouble(docId, valueToIndex);
+ public void putInt(int value) {
+ _indexWriter.putInt(value);
}
@Override
- public void index(int docId, Object valueToIndex) {
- if (valueToIndex instanceof Integer) {
- index(docId, ((Integer) valueToIndex).intValue());
- } else if (valueToIndex instanceof Long) {
- index(docId, ((Long) valueToIndex).longValue());
- } else if (valueToIndex instanceof Float) {
- index(docId, ((Float) valueToIndex).floatValue());
- } else if (valueToIndex instanceof Double) {
- index(docId, ((Double) valueToIndex).doubleValue());
- } else {
- throw new IllegalArgumentException(
- "Illegal argument type for fixed length raw indexing: " + valueToIndex.getClass().getName());
- }
+ public void putLong(long value) {
+ _indexWriter.putLong(value);
+ }
+
+ @Override
+ public void putFloat(float value) {
+ _indexWriter.putFloat(value);
+ }
+
+ @Override
+ public void putDouble(double value) {
+ _indexWriter.putDouble(value);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueSortedForwardIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueSortedForwardIndexCreator.java
index 21ce1a8..7049a39 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueSortedForwardIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueSortedForwardIndexCreator.java
@@ -23,15 +23,21 @@ import java.io.File;
import java.io.IOException;
import java.util.Arrays;
import org.apache.pinot.core.io.writer.impl.FixedByteSingleValueMultiColWriter;
-import org.apache.pinot.core.segment.creator.SingleValueForwardIndexCreator;
+import org.apache.pinot.core.segment.creator.ForwardIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.spi.data.FieldSpec;
-public class SingleValueSortedForwardIndexCreator implements SingleValueForwardIndexCreator {
+/**
+ * Forward index creator for dictionary-encoded sorted single-value column.
+ */
+public class SingleValueSortedForwardIndexCreator implements ForwardIndexCreator {
private final FixedByteSingleValueMultiColWriter _writer;
private final int[] _minDocIds;
private final int[] _maxDocIds;
+ private int _nextDocId = 0;
+
public SingleValueSortedForwardIndexCreator(File outputDir, String column, int cardinality)
throws Exception {
File indexFile = new File(outputDir, column + V1Constants.Indexes.SORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
@@ -44,7 +50,23 @@ public class SingleValueSortedForwardIndexCreator implements SingleValueForwardI
}
@Override
- public void index(int docId, int dictId) {
+ public boolean isDictionaryEncoded() {
+ return true;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return true;
+ }
+
+ @Override
+ public FieldSpec.DataType getValueType() {
+ return FieldSpec.DataType.INT;
+ }
+
+ @Override
+ public void putDictId(int dictId) {
+ int docId = _nextDocId++;
if (_minDocIds[dictId] > docId) {
_minDocIds[dictId] = docId;
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueUnsortedForwardIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueUnsortedForwardIndexCreator.java
index 76cc78c..17e1982 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueUnsortedForwardIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueUnsortedForwardIndexCreator.java
@@ -21,24 +21,42 @@ package org.apache.pinot.core.segment.creator.impl.fwd;
import java.io.File;
import java.io.IOException;
import org.apache.pinot.core.io.util.PinotDataBitSet;
-import org.apache.pinot.core.io.writer.SingleColumnSingleValueWriter;
-import org.apache.pinot.core.io.writer.impl.v1.FixedBitSingleValueWriter;
-import org.apache.pinot.core.segment.creator.SingleValueForwardIndexCreator;
+import org.apache.pinot.core.io.writer.impl.FixedBitSVForwardIndexWriter;
+import org.apache.pinot.core.segment.creator.ForwardIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.spi.data.FieldSpec;
-public class SingleValueUnsortedForwardIndexCreator implements SingleValueForwardIndexCreator {
- private final SingleColumnSingleValueWriter _writer;
+/**
+ * Forward index creator for dictionary-encoded unsorted single-value column.
+ */
+public class SingleValueUnsortedForwardIndexCreator implements ForwardIndexCreator {
+ private final FixedBitSVForwardIndexWriter _writer;
public SingleValueUnsortedForwardIndexCreator(File outputDir, String column, int cardinality, int numDocs)
throws Exception {
File indexFile = new File(outputDir, column + V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
- _writer = new FixedBitSingleValueWriter(indexFile, numDocs, PinotDataBitSet.getNumBitsPerValue(cardinality - 1));
+ _writer = new FixedBitSVForwardIndexWriter(indexFile, numDocs, PinotDataBitSet.getNumBitsPerValue(cardinality - 1));
+ }
+
+ @Override
+ public boolean isDictionaryEncoded() {
+ return true;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return true;
+ }
+
+ @Override
+ public FieldSpec.DataType getValueType() {
+ return FieldSpec.DataType.INT;
}
@Override
- public void index(int docId, int dictId) {
- _writer.setInt(docId, dictId);
+ public void putDictId(int dictId) {
+ _writer.putDictId(dictId);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueVarByteRawIndexCreator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueVarByteRawIndexCreator.java
index 9bb393c..086a74e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueVarByteRawIndexCreator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/fwd/SingleValueVarByteRawIndexCreator.java
@@ -22,17 +22,23 @@ import com.google.common.annotations.VisibleForTesting;
import java.io.File;
import java.io.IOException;
import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
-import org.apache.pinot.core.io.writer.impl.v1.BaseChunkSingleValueWriter;
-import org.apache.pinot.core.io.writer.impl.v1.VarByteChunkSingleValueWriter;
-import org.apache.pinot.core.segment.creator.BaseSingleValueRawIndexCreator;
+import org.apache.pinot.core.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.core.io.writer.impl.VarByteChunkSVForwardIndexWriter;
+import org.apache.pinot.core.segment.creator.ForwardIndexCreator;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
-public class SingleValueVarByteRawIndexCreator extends BaseSingleValueRawIndexCreator {
+/**
+ * Forward index creator for raw (non-dictionary-encoded) single-value column of variable length data type (STRING,
+ * BYTES).
+ */
+public class SingleValueVarByteRawIndexCreator implements ForwardIndexCreator {
private static final int DEFAULT_NUM_DOCS_PER_CHUNK = 1000;
private static final int TARGET_MAX_CHUNK_SIZE = 1024 * 1024;
- private final VarByteChunkSingleValueWriter _indexWriter;
+ private final VarByteChunkSVForwardIndexWriter _indexWriter;
+ private final DataType _valueType;
/**
* Create a var-byte raw index creator for the given column
@@ -40,14 +46,15 @@ public class SingleValueVarByteRawIndexCreator extends BaseSingleValueRawIndexCr
* @param compressionType Type of compression to use
* @param column Name of column to index
* @param totalDocs Total number of documents to index
+ * @param valueType Type of the values
* @param maxLength length of longest entry (in bytes)
* @throws IOException
*/
public SingleValueVarByteRawIndexCreator(File baseIndexDir, ChunkCompressorFactory.CompressionType compressionType,
- String column, int totalDocs, int maxLength)
+ String column, int totalDocs, DataType valueType, int maxLength)
throws IOException {
- this(baseIndexDir, compressionType, column, totalDocs, maxLength, false,
- BaseChunkSingleValueWriter.DEFAULT_VERSION);
+ this(baseIndexDir, compressionType, column, totalDocs, valueType, maxLength, false,
+ BaseChunkSVForwardIndexWriter.DEFAULT_VERSION);
}
/**
@@ -56,46 +63,51 @@ public class SingleValueVarByteRawIndexCreator extends BaseSingleValueRawIndexCr
* @param compressionType Type of compression to use
* @param column Name of column to index
* @param totalDocs Total number of documents to index
+ * @param valueType Type of the values
* @param maxLength length of longest entry (in bytes)
* @param deriveNumDocsPerChunk true if writer should auto-derive the number of rows per chunk
* @param writerVersion writer format version
* @throws IOException
*/
public SingleValueVarByteRawIndexCreator(File baseIndexDir, ChunkCompressorFactory.CompressionType compressionType,
- String column, int totalDocs, int maxLength, boolean deriveNumDocsPerChunk, int writerVersion)
+ String column, int totalDocs, DataType valueType, int maxLength, boolean deriveNumDocsPerChunk, int writerVersion)
throws IOException {
File file = new File(baseIndexDir, column + V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION);
int numDocsPerChunk = deriveNumDocsPerChunk ? getNumDocsPerChunk(maxLength) : DEFAULT_NUM_DOCS_PER_CHUNK;
- _indexWriter =
- new VarByteChunkSingleValueWriter(file, compressionType, totalDocs, numDocsPerChunk, maxLength, writerVersion);
+ _indexWriter = new VarByteChunkSVForwardIndexWriter(file, compressionType, totalDocs, numDocsPerChunk, maxLength,
+ writerVersion);
+ _valueType = valueType;
}
@VisibleForTesting
public static int getNumDocsPerChunk(int lengthOfLongestEntry) {
- int overheadPerEntry = lengthOfLongestEntry + VarByteChunkSingleValueWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE;
+ int overheadPerEntry = lengthOfLongestEntry + VarByteChunkSVForwardIndexWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE;
return Math.max(TARGET_MAX_CHUNK_SIZE / overheadPerEntry, 1);
}
@Override
- public void index(int docId, String valueToIndex) {
- _indexWriter.setString(docId, valueToIndex);
+ public boolean isDictionaryEncoded() {
+ return false;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return true;
+ }
+
+ @Override
+ public DataType getValueType() {
+ return _valueType;
}
@Override
- public void index(int docId, byte[] valueToIndex) {
- _indexWriter.setBytes(docId, valueToIndex);
+ public void putString(String value) {
+ _indexWriter.putString(value);
}
@Override
- public void index(int docId, Object valueToIndex) {
- if (valueToIndex instanceof String) {
- _indexWriter.setString(docId, (String) valueToIndex);
- } else if (valueToIndex instanceof byte[]) {
- _indexWriter.setBytes(docId, (byte[]) valueToIndex);
- } else {
- throw new IllegalArgumentException(
- "Illegal data type for variable length indexing: " + valueToIndex.getClass().getName());
- }
+ public void putBytes(byte[] value) {
+ _indexWriter.putBytes(value);
}
@Override
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/BaseVirtualColumnProvider.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/BaseVirtualColumnProvider.java
index 459ad35..41f97b5 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/BaseVirtualColumnProvider.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/BaseVirtualColumnProvider.java
@@ -40,6 +40,7 @@ public abstract class BaseVirtualColumnProvider implements VirtualColumnProvider
@Override
public ColumnIndexContainer buildColumnIndexContainer(VirtualColumnContext context) {
- return new VirtualColumnIndexContainer(buildReader(context), buildInvertedIndex(context), buildDictionary(context));
+ return new VirtualColumnIndexContainer(buildForwardIndex(context), buildInvertedIndex(context),
+ buildDictionary(context));
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/ColumnIndexContainer.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/ColumnIndexContainer.java
index 36578be..f0346d0 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/ColumnIndexContainer.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/ColumnIndexContainer.java
@@ -19,9 +19,9 @@
package org.apache.pinot.core.segment.index.column;
import java.io.Closeable;
-import org.apache.pinot.core.io.reader.DataFileReader;
import org.apache.pinot.core.segment.index.readers.BloomFilterReader;
import org.apache.pinot.core.segment.index.readers.Dictionary;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
import org.apache.pinot.core.segment.index.readers.NullValueVectorReaderImpl;
@@ -34,17 +34,17 @@ public interface ColumnIndexContainer extends Closeable {
/**
* Returns the forward index for the column.
*/
- DataFileReader getForwardIndex();
+ ForwardIndexReader<?> getForwardIndex();
/**
* Returns the inverted index for the column, or {@code null} if it does not exist.
*/
- InvertedIndexReader getInvertedIndex();
+ InvertedIndexReader<?> getInvertedIndex();
/**
* Returns the inverted index for the column, or {@code null} if it does not exist.
*/
- InvertedIndexReader getRangeIndex();
+ InvertedIndexReader<?> getRangeIndex();
/**
* Returns the dictionary for the column, or {@code null} if it does not exist.
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/DefaultNullValueVirtualColumnProvider.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/DefaultNullValueVirtualColumnProvider.java
index 6dd70b0..f7e9796 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/DefaultNullValueVirtualColumnProvider.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/DefaultNullValueVirtualColumnProvider.java
@@ -18,10 +18,6 @@
*/
package org.apache.pinot.core.segment.index.column;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.impl.ConstantMVForwardIndex;
-import org.apache.pinot.core.io.reader.impl.ConstantMVInvertedIndex;
-import org.apache.pinot.core.io.reader.impl.ConstantSVSortedIndex;
import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
import org.apache.pinot.core.segment.index.readers.ConstantValueBytesDictionary;
import org.apache.pinot.core.segment.index.readers.ConstantValueDoubleDictionary;
@@ -30,7 +26,11 @@ import org.apache.pinot.core.segment.index.readers.ConstantValueIntDictionary;
import org.apache.pinot.core.segment.index.readers.ConstantValueLongDictionary;
import org.apache.pinot.core.segment.index.readers.ConstantValueStringDictionary;
import org.apache.pinot.core.segment.index.readers.Dictionary;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
+import org.apache.pinot.core.segment.index.readers.constant.ConstantMVForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.constant.ConstantMVInvertedIndexReader;
+import org.apache.pinot.core.segment.index.readers.constant.ConstantSortedIndexReader;
import org.apache.pinot.core.segment.virtualcolumn.VirtualColumnContext;
import org.apache.pinot.spi.data.FieldSpec;
@@ -41,11 +41,11 @@ import org.apache.pinot.spi.data.FieldSpec;
public class DefaultNullValueVirtualColumnProvider extends BaseVirtualColumnProvider {
@Override
- public DataFileReader buildReader(VirtualColumnContext context) {
+ public ForwardIndexReader<?> buildForwardIndex(VirtualColumnContext context) {
if (context.getFieldSpec().isSingleValueField()) {
- return new ConstantSVSortedIndex(context.getTotalDocCount());
+ return new ConstantSortedIndexReader(context.getTotalDocCount());
} else {
- return new ConstantMVForwardIndex();
+ return new ConstantMVForwardIndexReader();
}
}
@@ -71,11 +71,11 @@ public class DefaultNullValueVirtualColumnProvider extends BaseVirtualColumnProv
}
@Override
- public InvertedIndexReader buildInvertedIndex(VirtualColumnContext context) {
+ public InvertedIndexReader<?> buildInvertedIndex(VirtualColumnContext context) {
if (context.getFieldSpec().isSingleValueField()) {
- return new ConstantSVSortedIndex(context.getTotalDocCount());
+ return new ConstantSortedIndexReader(context.getTotalDocCount());
} else {
- return new ConstantMVInvertedIndex(context.getTotalDocCount());
+ return new ConstantMVInvertedIndexReader(context.getTotalDocCount());
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/PhysicalColumnIndexContainer.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/PhysicalColumnIndexContainer.java
index e81366a..5eea7d4 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/PhysicalColumnIndexContainer.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/PhysicalColumnIndexContainer.java
@@ -21,14 +21,6 @@ package org.apache.pinot.core.segment.index.column;
import java.io.File;
import java.io.IOException;
import java.util.Map;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.core.io.reader.impl.v1.FixedBitMultiValueReader;
-import org.apache.pinot.core.io.reader.impl.v1.FixedBitSingleValueReader;
-import org.apache.pinot.core.io.reader.impl.v1.FixedByteChunkSingleValueReader;
-import org.apache.pinot.core.io.reader.impl.v1.SortedIndexReader;
-import org.apache.pinot.core.io.reader.impl.v1.SortedIndexReaderImpl;
-import org.apache.pinot.core.io.reader.impl.v1.VarByteChunkSingleValueReader;
import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
import org.apache.pinot.core.segment.index.readers.BaseImmutableDictionary;
@@ -37,6 +29,7 @@ import org.apache.pinot.core.segment.index.readers.BloomFilterReader;
import org.apache.pinot.core.segment.index.readers.BytesDictionary;
import org.apache.pinot.core.segment.index.readers.DoubleDictionary;
import org.apache.pinot.core.segment.index.readers.FloatDictionary;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
import org.apache.pinot.core.segment.index.readers.IntDictionary;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
import org.apache.pinot.core.segment.index.readers.LongDictionary;
@@ -47,7 +40,13 @@ import org.apache.pinot.core.segment.index.readers.OnHeapIntDictionary;
import org.apache.pinot.core.segment.index.readers.OnHeapLongDictionary;
import org.apache.pinot.core.segment.index.readers.OnHeapStringDictionary;
import org.apache.pinot.core.segment.index.readers.RangeIndexReader;
+import org.apache.pinot.core.segment.index.readers.SortedIndexReader;
import org.apache.pinot.core.segment.index.readers.StringDictionary;
+import org.apache.pinot.core.segment.index.readers.forward.FixedBitMVForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.forward.FixedBitSVForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.forward.FixedByteChunkSVForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.forward.VarByteChunkSVForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.sorted.SortedIndexReaderImpl;
import org.apache.pinot.core.segment.index.readers.text.LuceneTextIndexReader;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
import org.apache.pinot.core.segment.store.ColumnIndexType;
@@ -60,9 +59,9 @@ import org.slf4j.LoggerFactory;
public final class PhysicalColumnIndexContainer implements ColumnIndexContainer {
private static final Logger LOGGER = LoggerFactory.getLogger(PhysicalColumnIndexContainer.class);
- private final DataFileReader _forwardIndex;
- private final InvertedIndexReader _invertedIndex;
- private final InvertedIndexReader _rangeIndex;
+ private final ForwardIndexReader<?> _forwardIndex;
+ private final InvertedIndexReader<?> _invertedIndex;
+ private final InvertedIndexReader<?> _rangeIndex;
private final BaseImmutableDictionary _dictionary;
private final BloomFilterReader _bloomFilterReader;
private final NullValueVectorReaderImpl _nullValueVectorReader;
@@ -108,7 +107,7 @@ public final class PhysicalColumnIndexContainer implements ColumnIndexContainer
// Single-value
if (metadata.isSorted()) {
// Sorted
- SortedIndexReader sortedIndexReader = new SortedIndexReaderImpl(fwdIndexBuffer, metadata.getCardinality());
+ SortedIndexReader<?> sortedIndexReader = new SortedIndexReaderImpl(fwdIndexBuffer, metadata.getCardinality());
_forwardIndex = sortedIndexReader;
_invertedIndex = sortedIndexReader;
_rangeIndex = null;
@@ -116,13 +115,12 @@ public final class PhysicalColumnIndexContainer implements ColumnIndexContainer
} else {
// Unsorted
_forwardIndex =
- new FixedBitSingleValueReader(fwdIndexBuffer, metadata.getTotalDocs(), metadata.getBitsPerElement());
+ new FixedBitSVForwardIndexReader(fwdIndexBuffer, metadata.getTotalDocs(), metadata.getBitsPerElement());
}
} else {
// Multi-value
- _forwardIndex =
- new FixedBitMultiValueReader(fwdIndexBuffer, metadata.getTotalDocs(), metadata.getTotalNumberOfEntries(),
- metadata.getBitsPerElement());
+ _forwardIndex = new FixedBitMVForwardIndexReader(fwdIndexBuffer, metadata.getTotalDocs(),
+ metadata.getTotalNumberOfEntries(), metadata.getBitsPerElement());
}
if (loadInvertedIndex) {
_invertedIndex =
@@ -153,17 +151,17 @@ public final class PhysicalColumnIndexContainer implements ColumnIndexContainer
}
@Override
- public DataFileReader getForwardIndex() {
+ public ForwardIndexReader<?> getForwardIndex() {
return _forwardIndex;
}
@Override
- public InvertedIndexReader getInvertedIndex() {
+ public InvertedIndexReader<?> getInvertedIndex() {
return _invertedIndex;
}
@Override
- public InvertedIndexReader getRangeIndex() {
+ public InvertedIndexReader<?> getRangeIndex() {
return _rangeIndex;
}
@@ -224,18 +222,17 @@ public final class PhysicalColumnIndexContainer implements ColumnIndexContainer
}
}
- private static SingleColumnSingleValueReader loadRawForwardIndex(PinotDataBuffer forwardIndexBuffer,
+ private static ForwardIndexReader<?> loadRawForwardIndex(PinotDataBuffer forwardIndexBuffer,
FieldSpec.DataType dataType) {
-
switch (dataType) {
case INT:
case LONG:
case FLOAT:
case DOUBLE:
- return new FixedByteChunkSingleValueReader(forwardIndexBuffer);
+ return new FixedByteChunkSVForwardIndexReader(forwardIndexBuffer, dataType);
case STRING:
case BYTES:
- return new VarByteChunkSingleValueReader(forwardIndexBuffer);
+ return new VarByteChunkSVForwardIndexReader(forwardIndexBuffer, dataType);
default:
throw new IllegalStateException("Illegal data type for raw forward index: " + dataType);
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/BaseDataSource.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/BaseDataSource.java
index fc32487..4ad9528 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/BaseDataSource.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/BaseDataSource.java
@@ -19,35 +19,28 @@
package org.apache.pinot.core.segment.index.datasource;
import javax.annotation.Nullable;
-import org.apache.pinot.core.common.Block;
import org.apache.pinot.core.common.DataSource;
import org.apache.pinot.core.common.DataSourceMetadata;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.SingleColumnMultiValueReader;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.core.operator.blocks.MultiValueBlock;
-import org.apache.pinot.core.operator.blocks.SingleValueBlock;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
import org.apache.pinot.core.segment.index.readers.BloomFilterReader;
import org.apache.pinot.core.segment.index.readers.Dictionary;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
import org.apache.pinot.core.segment.index.readers.NullValueVectorReader;
-import org.apache.pinot.spi.data.FieldSpec;
-public abstract class BaseDataSource extends DataSource {
+public abstract class BaseDataSource implements DataSource {
private final DataSourceMetadata _dataSourceMetadata;
- private final DataFileReader _forwardIndex;
+ private final ForwardIndexReader<?> _forwardIndex;
private final Dictionary _dictionary;
- private final InvertedIndexReader _invertedIndex;
- private final InvertedIndexReader _rangeIndex;
+ private final InvertedIndexReader<?> _invertedIndex;
+ private final InvertedIndexReader<?> _rangeIndex;
private final BloomFilterReader _bloomFilter;
private final NullValueVectorReader _nullValueVector;
- private final String _operatorName;
- public BaseDataSource(DataSourceMetadata dataSourceMetadata, DataFileReader forwardIndex,
- @Nullable Dictionary dictionary, @Nullable InvertedIndexReader invertedIndex,
- @Nullable InvertedIndexReader rangeIndex, @Nullable BloomFilterReader bloomFilter,
- @Nullable NullValueVectorReader nullValueVector, String operatorName) {
+ public BaseDataSource(DataSourceMetadata dataSourceMetadata, ForwardIndexReader<?> forwardIndex,
+ @Nullable Dictionary dictionary, @Nullable InvertedIndexReader<?> invertedIndex,
+ @Nullable InvertedIndexReader<?> rangeIndex, @Nullable BloomFilterReader bloomFilter,
+ @Nullable NullValueVectorReader nullValueVector) {
_dataSourceMetadata = dataSourceMetadata;
_forwardIndex = forwardIndex;
_dictionary = dictionary;
@@ -55,7 +48,6 @@ public abstract class BaseDataSource extends DataSource {
_rangeIndex = rangeIndex;
_bloomFilter = bloomFilter;
_nullValueVector = nullValueVector;
- _operatorName = operatorName;
}
@Override
@@ -64,7 +56,7 @@ public abstract class BaseDataSource extends DataSource {
}
@Override
- public DataFileReader getForwardIndex() {
+ public ForwardIndexReader<?> getForwardIndex() {
return _forwardIndex;
}
@@ -76,13 +68,13 @@ public abstract class BaseDataSource extends DataSource {
@Nullable
@Override
- public InvertedIndexReader getInvertedIndex() {
+ public InvertedIndexReader<?> getInvertedIndex() {
return _invertedIndex;
}
@Nullable
@Override
- public InvertedIndexReader getRangeIndex() {
+ public InvertedIndexReader<?> getRangeIndex() {
return _rangeIndex;
}
@@ -97,21 +89,4 @@ public abstract class BaseDataSource extends DataSource {
public NullValueVectorReader getNullValueVector() {
return _nullValueVector;
}
-
- @Override
- protected Block getNextBlock() {
- FieldSpec.DataType dataType = _dataSourceMetadata.getDataType();
- if (_dataSourceMetadata.isSingleValue()) {
- return new SingleValueBlock((SingleColumnSingleValueReader) _forwardIndex, _dataSourceMetadata.getNumDocs(),
- dataType, _dictionary);
- } else {
- return new MultiValueBlock((SingleColumnMultiValueReader) _forwardIndex, _dataSourceMetadata.getNumDocs(),
- _dataSourceMetadata.getMaxNumValuesPerMVEntry(), dataType, _dictionary);
- }
- }
-
- @Override
- public String getOperatorName() {
- return _operatorName;
- }
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/ImmutableDataSource.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/ImmutableDataSource.java
index b4b0f5a..b4fb313 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/ImmutableDataSource.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/ImmutableDataSource.java
@@ -31,13 +31,12 @@ import org.apache.pinot.spi.data.FieldSpec;
* The {@code ImmutableDataSource} class is the data source for a column in the immutable segment.
*/
public class ImmutableDataSource extends BaseDataSource {
- private static final String OPERATOR_NAME_PREFIX = "ImmutableDataSource:";
public ImmutableDataSource(ColumnMetadata columnMetadata, ColumnIndexContainer columnIndexContainer) {
super(new ImmutableDataSourceMetadata(columnMetadata), columnIndexContainer.getForwardIndex(),
- columnIndexContainer.getDictionary(), columnIndexContainer.getInvertedIndex(), columnIndexContainer.getRangeIndex(),
- columnIndexContainer.getBloomFilter(), columnIndexContainer.getNullValueVector(),
- OPERATOR_NAME_PREFIX + columnMetadata.getColumnName());
+ columnIndexContainer.getDictionary(), columnIndexContainer.getInvertedIndex(),
+ columnIndexContainer.getRangeIndex(), columnIndexContainer.getBloomFilter(),
+ columnIndexContainer.getNullValueVector());
}
private static class ImmutableDataSourceMetadata implements DataSourceMetadata {
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/MutableDataSource.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/MutableDataSource.java
index 22f271e..48c235e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/MutableDataSource.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/datasource/MutableDataSource.java
@@ -23,7 +23,7 @@ import java.util.Set;
import javax.annotation.Nullable;
import org.apache.pinot.core.common.DataSourceMetadata;
import org.apache.pinot.core.data.partition.PartitionFunction;
-import org.apache.pinot.core.io.reader.DataFileReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
import org.apache.pinot.core.segment.index.readers.BloomFilterReader;
import org.apache.pinot.core.segment.index.readers.Dictionary;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
@@ -34,18 +34,16 @@ import org.apache.pinot.spi.data.FieldSpec;
/**
* The {@code MutableDataSource} class is the data source for a column in the mutable segment.
*/
-// TODO: Support min/max value for mutable segment
public class MutableDataSource extends BaseDataSource {
- private static final String OPERATOR_NAME_PREFIX = "MutableDataSource:";
public MutableDataSource(FieldSpec fieldSpec, int numDocs, int numValues, int maxNumValuesPerMVEntry,
@Nullable PartitionFunction partitionFunction, int partitionId, @Nullable Comparable minValue,
- @Nullable Comparable maxValue, DataFileReader forwardIndex, @Nullable Dictionary dictionary,
+ @Nullable Comparable maxValue, ForwardIndexReader forwardIndex, @Nullable Dictionary dictionary,
@Nullable InvertedIndexReader invertedIndex, @Nullable InvertedIndexReader rangeIndex,
@Nullable BloomFilterReader bloomFilter, @Nullable NullValueVectorReader nullValueVector) {
super(new MutableDataSourceMetadata(fieldSpec, numDocs, numValues, maxNumValuesPerMVEntry, partitionFunction,
- partitionId, minValue, maxValue), forwardIndex, dictionary, invertedIndex, rangeIndex, bloomFilter, nullValueVector,
- OPERATOR_NAME_PREFIX + fieldSpec.getName());
+ partitionId, minValue, maxValue), forwardIndex, dictionary, invertedIndex, rangeIndex, bloomFilter,
+ nullValueVector);
}
private static class MutableDataSourceMetadata implements DataSourceMetadata {
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java
index 9cbbea8..d25a97f 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java
@@ -21,7 +21,6 @@ package org.apache.pinot.core.segment.index.loader.defaultcolumn;
import com.google.common.base.Preconditions;
import java.io.File;
import java.io.IOException;
-import java.nio.charset.Charset;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
@@ -48,6 +47,7 @@ import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
import org.apache.pinot.core.segment.store.SegmentDirectory;
import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.utils.ByteArray;
import org.apache.pinot.spi.utils.BytesUtils;
@@ -129,8 +129,8 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
LoaderUtils.getStringListFromSegmentProperties(V1Constants.MetadataKeys.Segment.DIMENSIONS, _segmentProperties);
List<String> metricColumns =
LoaderUtils.getStringListFromSegmentProperties(V1Constants.MetadataKeys.Segment.METRICS, _segmentProperties);
- List<String> dateTimeColumns =
- LoaderUtils.getStringListFromSegmentProperties(V1Constants.MetadataKeys.Segment.DATETIME_COLUMNS, _segmentProperties);
+ List<String> dateTimeColumns = LoaderUtils
+ .getStringListFromSegmentProperties(V1Constants.MetadataKeys.Segment.DATETIME_COLUMNS, _segmentProperties);
for (Map.Entry<String, DefaultColumnAction> entry : defaultColumnActionMap.entrySet()) {
String column = entry.getKey();
DefaultColumnAction action = entry.getValue();
@@ -206,14 +206,14 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
}
// Check the data type and default value matches.
- FieldSpec.DataType dataTypeInMetadata = columnMetadata.getDataType();
- FieldSpec.DataType dataTypeInSchema = fieldSpecInSchema.getDataType();
+ DataType dataTypeInMetadata = columnMetadata.getDataType();
+ DataType dataTypeInSchema = fieldSpecInSchema.getDataType();
boolean isSingleValueInMetadata = columnMetadata.isSingleValue();
boolean isSingleValueInSchema = fieldSpecInSchema.isSingleValueField();
String defaultValueInMetadata = columnMetadata.getDefaultNullValueString();
String defaultValueInSchema;
- if (dataTypeInSchema == FieldSpec.DataType.BYTES) {
+ if (dataTypeInSchema == DataType.BYTES) {
defaultValueInSchema = BytesUtils.toHexString((byte[]) fieldSpecInSchema.getDefaultNullValue());
} else {
defaultValueInSchema = fieldSpecInSchema.getDefaultNullValue().toString();
@@ -227,7 +227,7 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
} else if (isSingleValueInMetadata != isSingleValueInSchema) {
defaultColumnActionMap.put(column, DefaultColumnAction.UPDATE_DIMENSION_NUMBER_OF_VALUES);
}
- } else if (fieldTypeInMetadata == FieldSpec.FieldType.METRIC){
+ } else if (fieldTypeInMetadata == FieldSpec.FieldType.METRIC) {
if (dataTypeInMetadata != dataTypeInSchema) {
defaultColumnActionMap.put(column, DefaultColumnAction.UPDATE_METRIC_DATA_TYPE);
} else if (!defaultValueInSchema.equals(defaultValueInMetadata)) {
@@ -235,7 +235,7 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
} else if (isSingleValueInMetadata != isSingleValueInSchema) {
defaultColumnActionMap.put(column, DefaultColumnAction.UPDATE_METRIC_NUMBER_OF_VALUES);
}
- } else if (fieldTypeInMetadata == FieldSpec.FieldType.DATE_TIME){
+ } else if (fieldTypeInMetadata == FieldSpec.FieldType.DATE_TIME) {
if (dataTypeInMetadata != dataTypeInSchema) {
defaultColumnActionMap.put(column, DefaultColumnAction.UPDATE_DATE_TIME_DATA_TYPE);
} else if (!defaultValueInSchema.equals(defaultValueInMetadata)) {
@@ -272,7 +272,7 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
FieldSpec.FieldType fieldTypeInMetadata = columnMetadata.getFieldType();
if (fieldTypeInMetadata == FieldSpec.FieldType.DIMENSION) {
defaultColumnActionMap.put(column, DefaultColumnAction.REMOVE_DIMENSION);
- } else if (fieldTypeInMetadata == FieldSpec.FieldType.METRIC){
+ } else if (fieldTypeInMetadata == FieldSpec.FieldType.METRIC) {
defaultColumnActionMap.put(column, DefaultColumnAction.REMOVE_METRIC);
} else if (fieldTypeInMetadata == FieldSpec.FieldType.DATE_TIME) {
defaultColumnActionMap.put(column, DefaultColumnAction.REMOVE_DATE_TIME);
@@ -345,7 +345,7 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
throw new UnsupportedOperationException("Text index is currently not supported on multi-value column: " + column);
}
- if (fieldSpec.getDataType() != FieldSpec.DataType.STRING) {
+ if (fieldSpec.getDataType() != DataType.STRING) {
throw new UnsupportedOperationException("Text index is currently only supported on STRING column:" + column);
}
}
@@ -357,27 +357,26 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
checkUnsupportedOperationsForTextIndex(column, indexLoadingConfig, fieldSpec);
int totalDocs = _segmentMetadata.getTotalDocs();
- Object defaultValue = fieldSpec.getDefaultNullValue();
- String stringDefaultValue = (String) defaultValue;
- int lengthOfLongestEntry = StringUtil.encodeUtf8(stringDefaultValue).length;
+ String defaultValue = (String) fieldSpec.getDefaultNullValue();
+ int lengthOfLongestEntry = StringUtil.encodeUtf8(defaultValue).length;
int dictionaryElementSize = 0;
- boolean deriveNumDocsPerChunk = SegmentColumnarIndexCreator.shouldDeriveNumDocsPerChunk(column, indexLoadingConfig.getColumnProperties());
- int writerVersion = SegmentColumnarIndexCreator.rawIndexWriterVersion(column, indexLoadingConfig.getColumnProperties());
- SingleValueVarByteRawIndexCreator rawIndexCreator =
- new SingleValueVarByteRawIndexCreator(_indexDir, ChunkCompressorFactory.CompressionType.SNAPPY, column,
- totalDocs, lengthOfLongestEntry, deriveNumDocsPerChunk, writerVersion);
-
- for (int docId = 0; docId < totalDocs; docId++) {
- rawIndexCreator.index(docId, defaultValue);
+ boolean deriveNumDocsPerChunk =
+ SegmentColumnarIndexCreator.shouldDeriveNumDocsPerChunk(column, indexLoadingConfig.getColumnProperties());
+ int writerVersion =
+ SegmentColumnarIndexCreator.rawIndexWriterVersion(column, indexLoadingConfig.getColumnProperties());
+ try (SingleValueVarByteRawIndexCreator rawIndexCreator = new SingleValueVarByteRawIndexCreator(_indexDir,
+ ChunkCompressorFactory.CompressionType.SNAPPY, column, totalDocs, DataType.STRING, lengthOfLongestEntry,
+ deriveNumDocsPerChunk, writerVersion)) {
+ for (int docId = 0; docId < totalDocs; docId++) {
+ rawIndexCreator.putString(defaultValue);
+ }
}
- rawIndexCreator.close();
-
// even though the column is sorted, we should pass it as false so that during
// TEXT_MATCH query time, when index reader is created, we create TextIndexReader
// and not SortedIndexReader.
- Object sortedArray = new String[]{(String) defaultValue};
+ Object sortedArray = new String[]{defaultValue};
DefaultColumnStatistics columnStatistics =
new DefaultColumnStatistics(defaultValue /* min */, defaultValue /* max */, sortedArray, false, totalDocs, 0);
@@ -403,7 +402,7 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
// Generate column index creation information.
final int totalDocs = _segmentMetadata.getTotalDocs();
- final FieldSpec.DataType dataType = fieldSpec.getDataType();
+ final DataType dataType = fieldSpec.getDataType();
final Object defaultValue = fieldSpec.getDefaultNullValue();
final boolean isSingleValue = fieldSpec.isSingleValueField();
final int maxNumberOfMultiValueElements = isSingleValue ? 0 : 1;
@@ -461,23 +460,23 @@ public abstract class BaseDefaultColumnHandler implements DefaultColumnHandler {
if (isSingleValue) {
// Single-value column.
- SingleValueSortedForwardIndexCreator svFwdIndexCreator =
- new SingleValueSortedForwardIndexCreator(_indexDir, fieldSpec.getName(), 1/*cardinality*/);
- for (int docId = 0; docId < totalDocs; docId++) {
- svFwdIndexCreator.index(docId, 0/*dictionaryId*/);
+ try (SingleValueSortedForwardIndexCreator svFwdIndexCreator = new SingleValueSortedForwardIndexCreator(_indexDir,
+ fieldSpec.getName(), 1/*cardinality*/)) {
+ for (int docId = 0; docId < totalDocs; docId++) {
+ svFwdIndexCreator.putDictId(0);
+ }
}
- svFwdIndexCreator.close();
} else {
// Multi-value column.
- MultiValueUnsortedForwardIndexCreator mvFwdIndexCreator =
- new MultiValueUnsortedForwardIndexCreator(_indexDir, fieldSpec.getName(), 1/*cardinality*/,
- totalDocs/*numDocs*/, totalDocs/*totalNumberOfValues*/);
- int[] dictIds = {0};
- for (int docId = 0; docId < totalDocs; docId++) {
- mvFwdIndexCreator.index(docId, dictIds);
+ try (
+ MultiValueUnsortedForwardIndexCreator mvFwdIndexCreator = new MultiValueUnsortedForwardIndexCreator(_indexDir,
+ fieldSpec.getName(), 1/*cardinality*/, totalDocs/*numDocs*/, totalDocs/*totalNumberOfValues*/)) {
+ int[] dictIds = {0};
+ for (int docId = 0; docId < totalDocs; docId++) {
+ mvFwdIndexCreator.putDictIdMV(dictIds);
+ }
}
- mvFwdIndexCreator.close();
}
// Add the column metadata information to the metadata properties.
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/InvertedIndexHandler.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/InvertedIndexHandler.java
index 36e3806..dda6e0b 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/InvertedIndexHandler.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/InvertedIndexHandler.java
@@ -24,16 +24,16 @@ import java.util.HashSet;
import java.util.Set;
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.indexsegment.generator.SegmentVersion;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.SingleColumnMultiValueReader;
-import org.apache.pinot.core.io.reader.impl.v1.FixedBitMultiValueReader;
-import org.apache.pinot.core.io.reader.impl.v1.FixedBitSingleValueReader;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
import org.apache.pinot.core.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
import org.apache.pinot.core.segment.index.loader.LoaderUtils;
import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
+import org.apache.pinot.core.segment.index.readers.forward.FixedBitMVForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.forward.FixedBitSVForwardIndexReader;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
import org.apache.pinot.core.segment.store.ColumnIndexType;
import org.apache.pinot.core.segment.store.SegmentDirectory;
@@ -41,6 +41,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+@SuppressWarnings({"rawtypes", "unchecked"})
public class InvertedIndexHandler {
private static final Logger LOGGER = LoggerFactory.getLogger(InvertedIndexHandler.class);
@@ -106,21 +107,18 @@ public class InvertedIndexHandler {
try (OffHeapBitmapInvertedIndexCreator creator = new OffHeapBitmapInvertedIndexCreator(_indexDir,
columnMetadata.getFieldSpec(), columnMetadata.getCardinality(), numDocs,
columnMetadata.getTotalNumberOfEntries())) {
- try (DataFileReader fwdIndex = getForwardIndexReader(columnMetadata, _segmentWriter)) {
+ try (ForwardIndexReader forwardIndexReader = getForwardIndexReader(columnMetadata, _segmentWriter);
+ ForwardIndexReaderContext readerContext = forwardIndexReader.createContext()) {
if (columnMetadata.isSingleValue()) {
// Single-value column.
-
- FixedBitSingleValueReader svFwdIndex = (FixedBitSingleValueReader) fwdIndex;
for (int i = 0; i < numDocs; i++) {
- creator.add(svFwdIndex.getInt(i));
+ creator.add(forwardIndexReader.getDictId(i, readerContext));
}
} else {
// Multi-value column.
-
- SingleColumnMultiValueReader mvFwdIndex = (SingleColumnMultiValueReader) fwdIndex;
int[] dictIds = new int[columnMetadata.getMaxNumberOfMultiValues()];
for (int i = 0; i < numDocs; i++) {
- int length = mvFwdIndex.getIntArray(i, dictIds);
+ int length = forwardIndexReader.getDictIdMV(i, dictIds, readerContext);
creator.add(dictIds, length);
}
}
@@ -139,15 +137,17 @@ public class InvertedIndexHandler {
LOGGER.info("Created inverted index for segment: {}, column: {}", _segmentName, column);
}
- private DataFileReader getForwardIndexReader(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter)
+ private ForwardIndexReader<?> getForwardIndexReader(ColumnMetadata columnMetadata,
+ SegmentDirectory.Writer segmentWriter)
throws IOException {
PinotDataBuffer buffer = segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.FORWARD_INDEX);
int numRows = columnMetadata.getTotalDocs();
int numBitsPerValue = columnMetadata.getBitsPerElement();
if (columnMetadata.isSingleValue()) {
- return new FixedBitSingleValueReader(buffer, numRows, numBitsPerValue);
+ return new FixedBitSVForwardIndexReader(buffer, numRows, numBitsPerValue);
} else {
- return new FixedBitMultiValueReader(buffer, numRows, columnMetadata.getTotalNumberOfEntries(), numBitsPerValue);
+ return new FixedBitMVForwardIndexReader(buffer, numRows, columnMetadata.getTotalNumberOfEntries(),
+ numBitsPerValue);
}
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/RangeIndexHandler.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/RangeIndexHandler.java
index e792986..4f1bce0 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/RangeIndexHandler.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/RangeIndexHandler.java
@@ -24,16 +24,16 @@ import java.util.HashSet;
import java.util.Set;
import org.apache.commons.io.FileUtils;
import org.apache.pinot.core.indexsegment.generator.SegmentVersion;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.SingleColumnMultiValueReader;
-import org.apache.pinot.core.io.reader.impl.v1.FixedBitMultiValueReader;
-import org.apache.pinot.core.io.reader.impl.v1.FixedBitSingleValueReader;
import org.apache.pinot.core.segment.creator.impl.V1Constants;
import org.apache.pinot.core.segment.creator.impl.inv.RangeIndexCreator;
import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
import org.apache.pinot.core.segment.index.loader.LoaderUtils;
import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
+import org.apache.pinot.core.segment.index.readers.forward.FixedBitMVForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.forward.FixedBitSVForwardIndexReader;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
import org.apache.pinot.core.segment.store.ColumnIndexType;
import org.apache.pinot.core.segment.store.SegmentDirectory;
@@ -42,6 +42,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+@SuppressWarnings({"rawtypes", "unchecked"})
public class RangeIndexHandler {
private static final Logger LOGGER = LoggerFactory.getLogger(RangeIndexHandler.class);
@@ -119,19 +120,18 @@ public class RangeIndexHandler {
int numDocs = columnMetadata.getTotalDocs();
try (RangeIndexCreator creator = new RangeIndexCreator(_indexDir, columnMetadata.getFieldSpec(),
FieldSpec.DataType.INT, -1, -1, numDocs, columnMetadata.getTotalNumberOfEntries())) {
- try (DataFileReader fwdIndex = getForwardIndexReader(columnMetadata, _segmentWriter)) {
+ try (ForwardIndexReader forwardIndexReader = getForwardIndexReader(columnMetadata, _segmentWriter);
+ ForwardIndexReaderContext readerContext = forwardIndexReader.createContext()) {
if (columnMetadata.isSingleValue()) {
// Single-value column.
- FixedBitSingleValueReader svFwdIndex = (FixedBitSingleValueReader) fwdIndex;
for (int i = 0; i < numDocs; i++) {
- creator.add(svFwdIndex.getInt(i));
+ creator.add(forwardIndexReader.getDictId(i, readerContext));
}
} else {
// Multi-value column.
- SingleColumnMultiValueReader mvFwdIndex = (SingleColumnMultiValueReader) fwdIndex;
int[] dictIds = new int[columnMetadata.getMaxNumberOfMultiValues()];
for (int i = 0; i < numDocs; i++) {
- int length = mvFwdIndex.getIntArray(i, dictIds);
+ int length = forwardIndexReader.getDictIdMV(i, dictIds, readerContext);
creator.add(dictIds, length);
}
}
@@ -140,15 +140,17 @@ public class RangeIndexHandler {
}
}
- private DataFileReader getForwardIndexReader(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter)
+ private ForwardIndexReader<?> getForwardIndexReader(ColumnMetadata columnMetadata,
+ SegmentDirectory.Writer segmentWriter)
throws IOException {
PinotDataBuffer buffer = segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.FORWARD_INDEX);
int numRows = columnMetadata.getTotalDocs();
int numBitsPerValue = columnMetadata.getBitsPerElement();
if (columnMetadata.isSingleValue()) {
- return new FixedBitSingleValueReader(buffer, numRows, numBitsPerValue);
+ return new FixedBitSVForwardIndexReader(buffer, numRows, numBitsPerValue);
} else {
- return new FixedBitMultiValueReader(buffer, numRows, columnMetadata.getTotalNumberOfEntries(), numBitsPerValue);
+ return new FixedBitMVForwardIndexReader(buffer, numRows, columnMetadata.getTotalNumberOfEntries(),
+ numBitsPerValue);
}
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/TextIndexHandler.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/TextIndexHandler.java
index a11b25c..6a37f7f 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/TextIndexHandler.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/TextIndexHandler.java
@@ -40,21 +40,19 @@ import java.io.File;
import java.io.IOException;
import java.util.HashSet;
import java.util.Set;
-import javax.annotation.Nonnull;
import org.apache.commons.configuration.PropertiesConfiguration;
import org.apache.pinot.core.indexsegment.generator.SegmentVersion;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.impl.ChunkReaderContext;
-import org.apache.pinot.core.io.reader.impl.v1.VarByteChunkSingleValueReader;
import org.apache.pinot.core.segment.creator.TextIndexType;
import org.apache.pinot.core.segment.creator.impl.inv.text.LuceneTextIndexCreator;
import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
import org.apache.pinot.core.segment.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.core.segment.index.readers.forward.BaseChunkSVForwardIndexReader.ChunkReaderContext;
+import org.apache.pinot.core.segment.index.readers.forward.VarByteChunkSVForwardIndexReader;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
import org.apache.pinot.core.segment.store.ColumnIndexType;
import org.apache.pinot.core.segment.store.SegmentDirectory;
import org.apache.pinot.core.segment.store.SegmentDirectoryPaths;
-import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -89,8 +87,8 @@ public class TextIndexHandler {
private final SegmentVersion _segmentVersion;
private final Set<ColumnMetadata> _textIndexColumns = new HashSet<>();
- public TextIndexHandler(@Nonnull File indexDir, @Nonnull SegmentMetadataImpl segmentMetadata,
- @Nonnull Set<String> textIndexColumns, @Nonnull SegmentDirectory.Writer segmentWriter) {
+ public TextIndexHandler(File indexDir, SegmentMetadataImpl segmentMetadata, Set<String> textIndexColumns,
+ SegmentDirectory.Writer segmentWriter) {
_indexDir = indexDir;
_segmentWriter = segmentWriter;
_segmentName = segmentMetadata.getName();
@@ -139,7 +137,7 @@ public class TextIndexHandler {
"Text index is currently not supported on multi-value columns: " + column);
}
- if (columnMetadata.getDataType() != FieldSpec.DataType.STRING) {
+ if (columnMetadata.getDataType() != DataType.STRING) {
throw new UnsupportedOperationException("Text index is currently only supported on STRING columns: " + column);
}
}
@@ -160,16 +158,13 @@ public class TextIndexHandler {
// segmentDirectory is indicated to us by SegmentDirectoryPaths, we create lucene index there. There is no
// further need to move around the lucene index directory since it is created with correct directory structure
// based on segmentVersion.
- try (LuceneTextIndexCreator textIndexCreator = new LuceneTextIndexCreator(column, segmentDirectory, true)) {
- try (DataFileReader forwardIndexReader = getForwardIndexReader(columnMetadata)) {
- VarByteChunkSingleValueReader forwardIndex = (VarByteChunkSingleValueReader) forwardIndexReader;
- ChunkReaderContext readerContext = forwardIndex.createContext();
- for (int docID = 0; docID < numDocs; docID++) {
- Object docToAdd = forwardIndex.getString(docID, readerContext);
- textIndexCreator.addDoc(docToAdd, docID);
- }
- textIndexCreator.seal();
+ try (LuceneTextIndexCreator textIndexCreator = new LuceneTextIndexCreator(column, segmentDirectory, true);
+ VarByteChunkSVForwardIndexReader forwardIndexReader = getForwardIndexReader(columnMetadata);
+ ChunkReaderContext readerContext = forwardIndexReader.createContext()) {
+ for (int docId = 0; docId < numDocs; docId++) {
+ textIndexCreator.addDoc(forwardIndexReader.getString(docId, readerContext), docId);
}
+ textIndexCreator.seal();
}
LOGGER.info("Created text index for column: {} in segment: {}", column, _segmentName);
PropertiesConfiguration properties = SegmentMetadataImpl.getPropertiesConfiguration(_indexDir);
@@ -177,9 +172,9 @@ public class TextIndexHandler {
properties.save();
}
- private DataFileReader getForwardIndexReader(ColumnMetadata columnMetadata)
+ private VarByteChunkSVForwardIndexReader getForwardIndexReader(ColumnMetadata columnMetadata)
throws IOException {
PinotDataBuffer buffer = _segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.FORWARD_INDEX);
- return new VarByteChunkSingleValueReader(buffer);
+ return new VarByteChunkSVForwardIndexReader(buffer, DataType.STRING);
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/BitmapInvertedIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/BitmapInvertedIndexReader.java
index 5e48913..5895571 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/BitmapInvertedIndexReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/BitmapInvertedIndexReader.java
@@ -88,13 +88,6 @@ public class BitmapInvertedIndexReader implements InvertedIndexReader<ImmutableR
}
}
- @Override
- public ImmutableRoaringBitmap getDocIds(Object value) {
- // This should not be called from anywhere. If it happens, there is a bug
- // and that's why we throw illegal state exception
- throw new IllegalStateException("bitmap inverted index reader supports lookup only on dictionary id");
- }
-
private synchronized ImmutableRoaringBitmap buildRoaringBitmapForIndex(final int index) {
int currentOffset = getOffset(index);
int bufferLength = getOffset(index + 1) - currentOffset;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/ForwardIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/ForwardIndexReader.java
new file mode 100644
index 0000000..1b613dc
--- /dev/null
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/ForwardIndexReader.java
@@ -0,0 +1,245 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.index.readers;
+
+import java.io.Closeable;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+
+
+/**
+ * Interface for forward index reader.
+ *
+ * @param <T> Type of the ReaderContext
+ */
+public interface ForwardIndexReader<T extends ForwardIndexReaderContext> extends Closeable {
+
+ /**
+ * Returns {@code true} if the forward index is dictionary-encoded, {@code false} if it is raw.
+ */
+ boolean isDictionaryEncoded();
+
+ /**
+ * Returns {@code true} if the forward index is for a single-value column, {@code false} if it is for a multi-value
+ * column.
+ */
+ boolean isSingleValue();
+
+ /**
+ * Returns the data type of the values in the forward index. Returns {@link DataType#INT} for dictionary-encoded
+ * forward index.
+ */
+ DataType getValueType();
+
+ /**
+ * Creates a new {@link ForwardIndexReaderContext} of the reader which can be used to accelerate the reads.
+ * NOTE: Caller is responsible for closing the returned reader context.
+ */
+ @Nullable
+ default T createContext() {
+ return null;
+ }
+
+ /**
+ * DICTIONARY-ENCODED INDEX APIs
+ */
+
+ /**
+ * Reads the dictionary id for a single-value column at the given document id.
+ *
+ * @param docId Document id
+ * @param context Reader context
+ * @return Dictionary id at the given document id
+ */
+ default int getDictId(int docId, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Batch reads multiple dictionary ids for a single-value column at the given document ids into the passed in buffer
+ * (the buffer size must be larger than or equal to the length).
+ *
+ * @param docIds Array containing the document ids to read
+ * @param length Number of values to read
+ * @param dictIdBuffer Dictionary id buffer
+ * @param context Reader context
+ */
+ default void readDictIds(int[] docIds, int length, int[] dictIdBuffer, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the dictionary ids for a multi-value column at the given document id into the passed in buffer (the buffer
+ * size must be enough to hold all the values for the multi-value entry) and returns the number of values within the
+ * multi-value entry.
+ *
+ * @param docId Document id
+ * @param dictIdBuffer Dictionary id buffer
+ * @param context Reader context
+ * @return Number of values within the multi-value entry
+ */
+ default int getDictIdMV(int docId, int[] dictIdBuffer, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * SINGLE-VALUE COLUMN RAW INDEX APIs
+ */
+
+ /**
+ * Reads the INT value at the given document id.
+ *
+ * @param docId Document id
+ * @param context Reader context
+ * @return INT type single-value at the given document id
+ */
+ default int getInt(int docId, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the LONG type single-value at the given document id.
+ *
+ * @param docId Document id
+ * @param context Reader context
+ * @return LONG type single-value at the given document id
+ */
+ default long getLong(int docId, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the FLOAT type single-value at the given document id.
+ *
+ * @param docId Document id
+ * @param context Reader context
+ * @return FLOAT type single-value at the given document id
+ */
+ default float getFloat(int docId, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the DOUBLE type single-value at the given document id.
+ *
+ * @param docId Document id
+ * @param context Reader context
+ * @return DOUBLE type single-value at the given document id
+ */
+ default double getDouble(int docId, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the STRING type single-value at the given document id.
+ *
+ * @param docId Document id
+ * @param context Reader context
+ * @return STRING type single-value at the given document id
+ */
+ default String getString(int docId, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the BYTES type single-value at the given document id.
+ *
+ * @param docId Document id
+ * @param context Reader context
+ * @return BYTES type single-value at the given document id
+ */
+ default byte[] getBytes(int docId, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * MULTI-VALUE COLUMN RAW INDEX APIs
+ * TODO: Not supported yet
+ */
+
+ /**
+ * Reads the INT type multi-value at the given document id into the passed in value buffer (the buffer size must be
+ * enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @param context Reader context
+ * @return Number of values within the multi-value entry
+ */
+ default int getIntMV(int docId, int[] valueBuffer, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the LONG type multi-value at the given document id into the passed in value buffer (the buffer size must be
+ * enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @param context Reader context
+ * @return Number of values within the multi-value entry
+ */
+ default int getLongMV(int docId, long[] valueBuffer, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the FLOAT type multi-value at the given document id into the passed in value buffer (the buffer size must be
+ * enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @param context Reader context
+ * @return Number of values within the multi-value entry
+ */
+ default int getFloatMV(int docId, float[] valueBuffer, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the DOUBLE type multi-value at the given document id into the passed in value buffer (the buffer size must
+ * be enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @param context Reader context
+ * @return Number of values within the multi-value entry
+ */
+ default int getDoubleMV(int docId, double[] valueBuffer, T context) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Reads the STRING type multi-value at the given document id into the passed in value buffer (the buffer size must
+ * be enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @param context Reader context
+ * @return Number of values within the multi-value entry
+ */
+ default int getStringMV(int docId, String[] valueBuffer, T context) {
+ throw new UnsupportedOperationException();
+ }
+}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/DataFileWriter.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/ForwardIndexReaderContext.java
similarity index 64%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/writer/DataFileWriter.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/ForwardIndexReaderContext.java
index debd6fc..9ed7b7d 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/writer/DataFileWriter.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/ForwardIndexReaderContext.java
@@ -16,15 +16,16 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.writer;
+package org.apache.pinot.core.segment.index.readers;
import java.io.Closeable;
/**
- *
- *
+ * Interface for the context of the forward index reader.
+ * <p>The forward index reader itself is always stateless because it needs to be accessed by multiple threads. The
+ * context can be passed when reading the values from the forward index reader so that reader can store some states
+ * inside the context in order to accelerate the following reads.
*/
-public interface DataFileWriter extends Closeable {
-
+public interface ForwardIndexReaderContext extends Closeable {
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/InvertedIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/InvertedIndexReader.java
index cf4ac0c..5f493f0 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/InvertedIndexReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/InvertedIndexReader.java
@@ -24,13 +24,14 @@ import java.io.Closeable;
public interface InvertedIndexReader<T> extends Closeable {
/**
- * Get the document ids for the given dictionary id.
- * @param dictId dictionary ID
+ * Returns the document ids for the given dictionary id.
*/
T getDocIds(int dictId);
/**
- * Get the document ids for a given value
+ * Returns the document ids for the given string representation of the value.
*/
- T getDocIds(Object value);
+ default T getDocIds(String value) {
+ throw new UnsupportedOperationException();
+ }
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/MutableForwardIndex.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/MutableForwardIndex.java
new file mode 100644
index 0000000..fca9d55
--- /dev/null
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/MutableForwardIndex.java
@@ -0,0 +1,419 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.index.readers;
+
+/**
+ * Interface for mutable forward index (for CONSUMING segment).
+ * NOTE: Mutable forward index does not use reader context to accelerate the reads.
+ */
+public interface MutableForwardIndex extends ForwardIndexReader<ForwardIndexReaderContext> {
+
+ /**
+ * Returns the length (size in bytes) of the shortest elements inside the forward index.
+ *
+ * @return The length (size in bytes) of the shortest elements inside the forward index.
+ */
+ int getLengthOfShortestElement();
+
+ /**
+ * Returns the length (size in bytes) of the longest elements inside the forward index.
+ *
+ * @return The length (size in bytes) of the longest elements inside the forward index.
+ */
+ int getLengthOfLongestElement();
+
+ /**
+ * DICTIONARY-ENCODED INDEX APIs
+ */
+
+ /**
+ * Reads the dictionary id for a single-value column at the given document id.
+ *
+ * @param docId Document id
+ * @return Dictionary id at the given document id
+ */
+ default int getDictId(int docId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default int getDictId(int docId, ForwardIndexReaderContext context) {
+ return getDictId(docId);
+ }
+
+ /**
+ * Batch reads multiple dictionary ids for a single-value column at the given document ids into the passed in buffer
+ * (the buffer size must be larger than or equal to the length).
+ *
+ * @param docIds Array containing the document ids to read
+ * @param length Number of values to read
+ * @param dictIdBuffer Dictionary id buffer
+ */
+ default void readDictIds(int[] docIds, int length, int[] dictIdBuffer) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default void readDictIds(int[] docIds, int length, int[] dictIdBuffer, ForwardIndexReaderContext context) {
+ readDictIds(docIds, length, dictIdBuffer);
+ }
+
+ /**
+ * Reads the dictionary ids for a multi-value column at the given document id into the passed in buffer (the buffer
+ * size must be enough to hold all the values for the multi-value entry) and returns the number of values within the
+ * multi-value entry.
+ *
+ * @param docId Document id
+ * @param dictIdBuffer Dictionary id buffer
+ * @return Number of values within the multi-value entry
+ */
+ default int getDictIdMV(int docId, int[] dictIdBuffer) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default int getDictIdMV(int docId, int[] dictIdBuffer, ForwardIndexReaderContext context) {
+ return getDictIdMV(docId, dictIdBuffer);
+ }
+
+ /**
+ * Writes the dictionary id for a single-value column into the given document id.
+ *
+ * @param docId Document id
+ * @param dictId Dictionary id to write
+ */
+ default void setDictId(int docId, int dictId) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the dictionary ids for a multi-value column into the given document id.
+ *
+ * @param docId Document id
+ * @param dictIds Dictionary ids to write
+ */
+ default void setDictIdMV(int docId, int[] dictIds) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * SINGLE-VALUE COLUMN RAW INDEX APIs
+ */
+
+ /**
+ * Reads the INT value at the given document id. The passed in reader context can be used to accelerate the reads.
+ * <p>NOTE: Dictionary id is handled as INT type.
+ *
+ * @param docId Document id
+ * @return INT type single-value at the given document id
+ */
+ default int getInt(int docId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default int getInt(int docId, ForwardIndexReaderContext context) {
+ return getInt(docId);
+ }
+
+ /**
+ * Reads the LONG type single-value at the given document id. The passed in reader context can be used to accelerate
+ * the reads.
+ *
+ * @param docId Document id
+ * @return LONG type single-value at the given document id
+ */
+ default long getLong(int docId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default long getLong(int docId, ForwardIndexReaderContext context) {
+ return getLong(docId);
+ }
+
+ /**
+ * Reads the FLOAT type single-value at the given document id. The passed in reader context can be used to accelerate
+ * the reads.
+ *
+ * @param docId Document id
+ * @return FLOAT type single-value at the given document id
+ */
+ default float getFloat(int docId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default float getFloat(int docId, ForwardIndexReaderContext context) {
+ return getFloat(docId);
+ }
+
+ /**
+ * Reads the DOUBLE type single-value at the given document id. The passed in reader context can be used to accelerate
+ * the reads.
+ *
+ * @param docId Document id
+ * @return DOUBLE type single-value at the given document id
+ */
+ default double getDouble(int docId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default double getDouble(int docId, ForwardIndexReaderContext context) {
+ return getDouble(docId);
+ }
+
+ /**
+ * Reads the STRING type single-value at the given document id. The passed in reader context can be used to accelerate
+ * the reads.
+ *
+ * @param docId Document id
+ * @return STRING type single-value at the given document id
+ */
+ default String getString(int docId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default String getString(int docId, ForwardIndexReaderContext context) {
+ return getString(docId);
+ }
+
+ /**
+ * Reads the BYTES type single-value at the given document id. The passed in reader context can be used to accelerate
+ * the reads.
+ *
+ * @param docId Document id
+ * @return BYTES type single-value at the given document id
+ */
+ default byte[] getBytes(int docId) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default byte[] getBytes(int docId, ForwardIndexReaderContext context) {
+ return getBytes(docId);
+ }
+
+ /**
+ * Writes the INT type single-value into the given document id.
+ *
+ * @param docId Document id
+ * @param value Value to write
+ */
+ default void setInt(int docId, int value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the LONG type single-value into the given document id.
+ *
+ * @param docId Document id
+ * @param value Value to write
+ */
+ default void setLong(int docId, long value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the FLOAT type single-value into the given document id.
+ *
+ * @param docId Document id
+ * @param value Value to write
+ */
+ default void setFloat(int docId, float value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the DOUBLE type single-value into the given document id.
+ *
+ * @param docId Document id
+ * @param value Value to write
+ */
+ default void setDouble(int docId, double value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the STRING type single-value into the given document id.
+ *
+ * @param docId Document id
+ * @param value Value to write
+ */
+ default void setString(int docId, String value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the BYTES type single-value into the given document id.
+ *
+ * @param docId Document id
+ * @param value Value to write
+ */
+ default void setBytes(int docId, byte[] value) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * MULTI-VALUE COLUMN RAW INDEX APIs
+ * TODO: Not supported yet
+ */
+
+ /**
+ * Reads the INT type multi-value at the given document id into the passed in value buffer (the buffer size must be
+ * enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @return Number of values within the multi-value entry
+ */
+ default int getIntMV(int docId, int[] valueBuffer) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default int getIntMV(int docId, int[] valueBuffer, ForwardIndexReaderContext context) {
+ return getIntMV(docId, valueBuffer);
+ }
+
+ /**
+ * Reads the LONG type multi-value at the given document id into the passed in value buffer (the buffer size must be
+ * enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @return Number of values within the multi-value entry
+ */
+ default int getLongMV(int docId, long[] valueBuffer) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default int getLongMV(int docId, long[] valueBuffer, ForwardIndexReaderContext context) {
+ return getLongMV(docId, valueBuffer);
+ }
+
+ /**
+ * Reads the FLOAT type multi-value at the given document id into the passed in value buffer (the buffer size must be
+ * enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @return Number of values within the multi-value entry
+ */
+ default int getFloatMV(int docId, float[] valueBuffer) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default int getFloatMV(int docId, float[] valueBuffer, ForwardIndexReaderContext context) {
+ return getFloatMV(docId, valueBuffer);
+ }
+
+ /**
+ * Reads the DOUBLE type multi-value at the given document id into the passed in value buffer (the buffer size must
+ * be enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @return Number of values within the multi-value entry
+ */
+ default int getDoubleMV(int docId, double[] valueBuffer) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default int getDoubleMV(int docId, double[] valueBuffer, ForwardIndexReaderContext context) {
+ return getDoubleMV(docId, valueBuffer);
+ }
+
+ /**
+ * Reads the STRING type multi-value at the given document id into the passed in value buffer (the buffer size must
+ * be enough to hold all the values for the multi-value entry) and returns the number of values within the multi-value
+ * entry.
+ *
+ * @param docId Document id
+ * @param valueBuffer Value buffer
+ * @return Number of values within the multi-value entry
+ */
+ default int getStringMV(int docId, String[] valueBuffer) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ default int getStringMV(int docId, String[] valueBuffer, ForwardIndexReaderContext context) {
+ return getStringMV(docId, valueBuffer);
+ }
+
+ /**
+ * Writes the INT type multi-value into the given document id.
+ *
+ * @param docId Document id
+ * @param values Values to write
+ */
+ default void setIntMV(int docId, int[] values) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the LONG type multi-value into the given document id.
+ *
+ * @param docId Document id
+ * @param values Values to write
+ */
+ default void setLongMV(int docId, long[] values) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the FLOAT type multi-value into the given document id.
+ *
+ * @param docId Document id
+ * @param values Values to write
+ */
+ default void setFloatMV(int docId, float[] values) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the DOUBLE type multi-value into the given document id.
+ *
+ * @param docId Document id
+ * @param values Values to write
+ */
+ default void setDoubleMV(int docId, double[] values) {
+ throw new UnsupportedOperationException();
+ }
+
+ /**
+ * Writes the STRING type multi-value into the given document id.
+ *
+ * @param docId Document id
+ * @param values Values to write
+ */
+ default void setStringMV(int docId, String[] values) {
+ throw new UnsupportedOperationException();
+ }
+}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/RangeIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/RangeIndexReader.java
index 4a57c9c..1f295af 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/RangeIndexReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/RangeIndexReader.java
@@ -141,13 +141,6 @@ public class RangeIndexReader implements InvertedIndexReader<ImmutableRoaringBit
}
}
- @Override
- public ImmutableRoaringBitmap getDocIds(Object value) {
- // This should not be called from anywhere. If it happens, there is a bug
- // and that's why we throw illegal state exception
- throw new IllegalStateException("bitmap inverted index reader supports lookup only on dictionary id");
- }
-
private synchronized ImmutableRoaringBitmap buildRoaringBitmapForIndex(final int rangeId) {
final long currentOffset = getOffset(rangeId);
final long nextOffset = getOffset(rangeId + 1);
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantMVForwardIndex.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/SortedIndexReader.java
similarity index 55%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantMVForwardIndex.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/SortedIndexReader.java
index 1faa904..fc13415 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantMVForwardIndex.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/SortedIndexReader.java
@@ -16,37 +16,35 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.reader.impl;
+package org.apache.pinot.core.segment.index.readers;
-import java.io.IOException;
-import org.apache.pinot.core.io.reader.BaseSingleColumnMultiValueReader;
-import org.apache.pinot.core.io.reader.ReaderContext;
+import org.apache.pinot.common.utils.Pairs;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
/**
- * Forward index for multi-value column with constant values.
+ * Interface for sorted index reader which can be used as both forward index and inverted index.
*/
-public class ConstantMVForwardIndex extends BaseSingleColumnMultiValueReader<ReaderContext> {
+public interface SortedIndexReader<T extends ForwardIndexReaderContext> extends ForwardIndexReader<T>, InvertedIndexReader<Pairs.IntPair> {
+ /**
+ * NOTE: Sorted index is always dictionary-encoded.
+ */
@Override
- public int getIntArray(int row, int[] intArray) {
- intArray[0] = 0;
- return 1;
+ default boolean isDictionaryEncoded() {
+ return true;
}
+ /**
+ * NOTE: Sorted index can only apply to single-value column.
+ */
@Override
- public int getIntArray(int row, int[] intArray, ReaderContext context) {
- intArray[0] = 0;
- return 1;
+ default boolean isSingleValue() {
+ return true;
}
@Override
- public ReaderContext createContext() {
- return null;
- }
-
- @Override
- public void close()
- throws IOException {
+ default DataType getValueType() {
+ return DataType.INT;
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/MultiValueSet.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantMVForwardIndexReader.java
similarity index 56%
rename from pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/MultiValueSet.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantMVForwardIndexReader.java
index 1ce6cca..6f67f76 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/operator/docvalsets/MultiValueSet.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantMVForwardIndexReader.java
@@ -16,38 +16,40 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.operator.docvalsets;
+package org.apache.pinot.core.segment.index.readers.constant;
-import org.apache.pinot.core.common.BaseBlockValSet;
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.reader.SingleColumnMultiValueReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
import org.apache.pinot.spi.data.FieldSpec.DataType;
-@SuppressWarnings({"rawtypes", "unchecked"})
-public final class MultiValueSet extends BaseBlockValSet {
- private final SingleColumnMultiValueReader _reader;
- private final ReaderContext _readerContext;
- private final DataType _dataType;
+/**
+ * Dictionary-encoded forward index reader for multi-value column with constant values.
+ */
+public final class ConstantMVForwardIndexReader implements ForwardIndexReader<ForwardIndexReaderContext> {
+
+ @Override
+ public boolean isDictionaryEncoded() {
+ return true;
+ }
- public MultiValueSet(SingleColumnMultiValueReader reader, DataType dataType) {
- _reader = reader;
- _readerContext = reader.createContext();
- _dataType = dataType;
+ @Override
+ public boolean isSingleValue() {
+ return false;
}
@Override
public DataType getValueType() {
- return _dataType;
+ return DataType.INT;
}
@Override
- public boolean isSingleValue() {
- return false;
+ public int getDictIdMV(int docId, int[] dictIdBuffer, ForwardIndexReaderContext context) {
+ dictIdBuffer[0] = 0;
+ return 1;
}
@Override
- public int getIntValues(int docId, int[] valueBuffer) {
- return _reader.getIntArray(docId, valueBuffer, _readerContext);
+ public void close() {
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantMVInvertedIndex.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantMVInvertedIndexReader.java
similarity index 70%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantMVInvertedIndex.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantMVInvertedIndexReader.java
index 40aee8c..b90d151 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/ConstantMVInvertedIndex.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantMVInvertedIndexReader.java
@@ -16,22 +16,21 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.reader.impl;
+package org.apache.pinot.core.segment.index.readers.constant;
-import java.io.IOException;
import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
import org.roaringbitmap.buffer.MutableRoaringBitmap;
/**
- * Inverted index for multi-value column with constant values.
+ * Inverted index reader for multi-value column with constant values.
*/
-public class ConstantMVInvertedIndex implements InvertedIndexReader<MutableRoaringBitmap> {
+public final class ConstantMVInvertedIndexReader implements InvertedIndexReader<MutableRoaringBitmap> {
private final MutableRoaringBitmap _bitmap;
- public ConstantMVInvertedIndex(int numDocs) {
+ public ConstantMVInvertedIndexReader(int numDocs) {
_bitmap = new MutableRoaringBitmap();
- _bitmap.add(0, numDocs);
+ _bitmap.add(0L, numDocs);
}
@Override
@@ -40,12 +39,6 @@ public class ConstantMVInvertedIndex implements InvertedIndexReader<MutableRoari
}
@Override
- public MutableRoaringBitmap getDocIds(Object value) {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void close()
- throws IOException {
+ public void close() {
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/SortedIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantSortedIndexReader.java
similarity index 51%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/SortedIndexReader.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantSortedIndexReader.java
index ce99a17..b1f1ef5 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/SortedIndexReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/constant/ConstantSortedIndexReader.java
@@ -16,35 +16,41 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.reader.impl.v1;
+package org.apache.pinot.core.segment.index.readers.constant;
-import java.io.IOException;
import org.apache.pinot.common.utils.Pairs;
-import org.apache.pinot.core.io.reader.ReaderContext;
-import org.apache.pinot.core.io.reader.SingleColumnSingleValueReader;
-import org.apache.pinot.core.segment.index.readers.InvertedIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
+import org.apache.pinot.core.segment.index.readers.SortedIndexReader;
/**
- * Interface for sorted index readers.
+ * Sorted index reader for single-value column with constant values.
*/
-public interface SortedIndexReader<T extends ReaderContext> extends SingleColumnSingleValueReader<T>, InvertedIndexReader<Pairs.IntPair> {
- @Override
- int getInt(int row);
+public final class ConstantSortedIndexReader implements SortedIndexReader<ForwardIndexReaderContext> {
+ private final int _numDocs;
- @Override
- int getInt(int row, T context);
+ public ConstantSortedIndexReader(int numDocs) {
+ _numDocs = numDocs;
+ }
@Override
- void readValues(int[] rows, int rowsStartIndex, int rowSize, int[] values, int valuesStartIndex);
+ public int getDictId(int docId, ForwardIndexReaderContext context) {
+ return 0;
+ }
@Override
- T createContext();
+ public void readDictIds(int[] docIds, int length, int[] dictIdBuffer, ForwardIndexReaderContext context) {
+ for (int i = 0; i < length; i++) {
+ dictIdBuffer[i] = 0;
+ }
+ }
@Override
- Pairs.IntPair getDocIds(int dictId);
+ public Pairs.IntPair getDocIds(int dictId) {
+ return new Pairs.IntPair(0, _numDocs - 1);
+ }
@Override
- void close()
- throws IOException;
+ public void close() {
+ }
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/BaseChunkSingleValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/BaseChunkSVForwardIndexReader.java
similarity index 64%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/BaseChunkSingleValueReader.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/BaseChunkSVForwardIndexReader.java
index 994e9ba..a573a4f 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/BaseChunkSingleValueReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/BaseChunkSVForwardIndexReader.java
@@ -16,42 +16,45 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.reader.impl.v1;
+package org.apache.pinot.core.segment.index.readers.forward;
+import com.google.common.base.Preconditions;
import java.io.IOException;
import java.nio.ByteBuffer;
import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
import org.apache.pinot.core.io.compression.ChunkDecompressor;
-import org.apache.pinot.core.io.reader.BaseSingleColumnSingleValueReader;
-import org.apache.pinot.core.io.reader.impl.ChunkReaderContext;
-import org.apache.pinot.core.io.writer.impl.v1.BaseChunkSingleValueWriter;
+import org.apache.pinot.core.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+import org.apache.pinot.core.util.CleanerUtil;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Abstract class implementation for {@link BaseSingleColumnSingleValueReader}.
- * Base class for the fixed and variable byte reader implementations.
- *
+ * Base implementation for chunk-based single-value raw (non-dictionary-encoded) forward index reader.
*/
-public abstract class BaseChunkSingleValueReader extends BaseSingleColumnSingleValueReader<ChunkReaderContext> {
- private static final Logger LOGGER = LoggerFactory.getLogger(BaseChunkSingleValueReader.class);
+public abstract class BaseChunkSVForwardIndexReader implements ForwardIndexReader<BaseChunkSVForwardIndexReader.ChunkReaderContext> {
+ private static final Logger LOGGER = LoggerFactory.getLogger(BaseChunkSVForwardIndexReader.class);
protected final int _chunkSize;
protected final int _numDocsPerChunk;
protected final int _numChunks;
protected final int _lengthOfLongestEntry;
+ protected final boolean _isCompressed;
+ protected final PinotDataBuffer _rawData;
private final PinotDataBuffer _dataBuffer;
+ private final DataType _valueType;
private final PinotDataBuffer _dataHeader;
- private final PinotDataBuffer _rawData;
- private final boolean _isCompressed;
private final ChunkDecompressor _chunkDecompressor;
private final int _headerEntryChunkOffsetSize;
- public BaseChunkSingleValueReader(PinotDataBuffer dataBuffer) {
+ public BaseChunkSVForwardIndexReader(PinotDataBuffer dataBuffer, DataType valueType) {
_dataBuffer = dataBuffer;
+ _valueType = valueType;
int headerOffset = 0;
int version = _dataBuffer.getInt(headerOffset);
@@ -64,6 +67,9 @@ public abstract class BaseChunkSingleValueReader extends BaseSingleColumnSingleV
headerOffset += Integer.BYTES;
_lengthOfLongestEntry = _dataBuffer.getInt(headerOffset);
+ if (valueType.isFixedWidth()) {
+ Preconditions.checkState(_lengthOfLongestEntry == valueType.size());
+ }
headerOffset += Integer.BYTES;
int dataHeaderStart = headerOffset;
@@ -84,7 +90,7 @@ public abstract class BaseChunkSingleValueReader extends BaseSingleColumnSingleV
}
_chunkSize = (_lengthOfLongestEntry * _numDocsPerChunk);
- _headerEntryChunkOffsetSize = BaseChunkSingleValueWriter.getHeaderEntryChunkOffsetSize(version);
+ _headerEntryChunkOffsetSize = BaseChunkSVForwardIndexWriter.getHeaderEntryChunkOffsetSize(version);
// Slice out the header from the data buffer.
int dataHeaderLength = _numChunks * _headerEntryChunkOffsetSize;
@@ -96,17 +102,17 @@ public abstract class BaseChunkSingleValueReader extends BaseSingleColumnSingleV
}
/**
- * Helper method to get the chunk for a given row.
+ * Helper method to return the chunk buffer that contains the value at the given document id.
* <ul>
* <li> If the chunk already exists in the reader context, returns the same. </li>
* <li> Otherwise, loads the chunk for the row, and sets it in the reader context. </li>
* </ul>
- * @param row Row for which to get the chunk
+ * @param docId Document id
* @param context Reader context
* @return Chunk for the row
*/
- protected ByteBuffer getChunkForRow(int row, ChunkReaderContext context) {
- int chunkId = row / _numDocsPerChunk;
+ protected ByteBuffer getChunkBuffer(int docId, ChunkReaderContext context) {
+ int chunkId = docId / _numDocsPerChunk;
if (context.getChunkId() == chunkId) {
return context.getChunkBuffer();
}
@@ -140,7 +146,7 @@ public abstract class BaseChunkSingleValueReader extends BaseSingleColumnSingleV
* @param chunkId Id of the chunk for which to return the position.
* @return Position (offset) of the chunk in the data.
*/
- protected long getChunkPosition(int chunkId) {
+ private long getChunkPosition(int chunkId) {
if (_headerEntryChunkOffsetSize == Integer.BYTES) {
return _dataHeader.getInt(chunkId * _headerEntryChunkOffsetSize);
} else {
@@ -148,22 +154,19 @@ public abstract class BaseChunkSingleValueReader extends BaseSingleColumnSingleV
}
}
- /**
- * Method to determine if the data is compressed or not.
- *
- * @return True if data is compressed, false otherwise.
- */
- protected boolean isCompressed() {
- return _isCompressed;
+ @Override
+ public boolean isDictionaryEncoded() {
+ return false;
}
- /**
- * Returns a PinotDataBuffer containing the raw data.
- *
- * @return PinotDataBuffer containing raw data.
- */
- protected PinotDataBuffer getRawData() {
- return _rawData;
+ @Override
+ public boolean isSingleValue() {
+ return true;
+ }
+
+ @Override
+ public DataType getValueType() {
+ return _valueType;
}
@Override
@@ -171,4 +174,45 @@ public abstract class BaseChunkSingleValueReader extends BaseSingleColumnSingleV
// NOTE: DO NOT close the PinotDataBuffer here because it is tracked by the caller and might be reused later. The
// caller is responsible of closing the PinotDataBuffer.
}
+
+ /**
+ * Context for the chunk-based forward index readers.
+ * <p>Information saved in the context can be used by subsequent reads as cache:
+ * <ul>
+ * <li>
+ * Chunk Buffer from the previous read. Useful if the subsequent read is from the same buffer, as it avoids extra
+ * chunk decompression.
+ * </li>
+ * <li>Id for the chunk</li>
+ * </ul>
+ */
+ public static class ChunkReaderContext implements ForwardIndexReaderContext {
+ private final ByteBuffer _chunkBuffer;
+ private int _chunkId;
+
+ public ChunkReaderContext(int maxChunkSize) {
+ _chunkBuffer = ByteBuffer.allocateDirect(maxChunkSize);
+ _chunkId = -1;
+ }
+
+ public ByteBuffer getChunkBuffer() {
+ return _chunkBuffer;
+ }
+
+ public int getChunkId() {
+ return _chunkId;
+ }
+
+ public void setChunkId(int chunkId) {
+ _chunkId = chunkId;
+ }
+
+ @Override
+ public void close()
+ throws IOException {
+ if (CleanerUtil.UNMAP_SUPPORTED) {
+ CleanerUtil.getCleaner().freeBuffer(_chunkBuffer);
+ }
+ }
+ }
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedBitMultiValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedBitMVForwardIndexReader.java
similarity index 65%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedBitMultiValueReader.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedBitMVForwardIndexReader.java
index 9f62a96..df6680b 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/FixedBitMultiValueReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedBitMVForwardIndexReader.java
@@ -16,18 +16,21 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.reader.impl.v1;
+package org.apache.pinot.core.segment.index.readers.forward;
-import org.apache.pinot.core.io.reader.BaseSingleColumnMultiValueReader;
-import org.apache.pinot.core.io.reader.ReaderContext;
import org.apache.pinot.core.io.util.FixedBitIntReaderWriter;
import org.apache.pinot.core.io.util.FixedByteValueReaderWriter;
import org.apache.pinot.core.io.util.PinotDataBitSet;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
/**
- * Storage Layout:
+ * Bit-compressed dictionary-encoded forward index reader for multi-value columns. The values returned are dictionary
+ * ids.
+ * <p>Storage Layout:
* <ul>
* <li>
* There will be three sections: CHUNK OFFSET, BITMAP and RAW DATA.
@@ -45,21 +48,21 @@ import org.apache.pinot.core.segment.memory.PinotDataBuffer;
* </li>
* </ul>
*/
-public final class FixedBitMultiValueReader extends BaseSingleColumnMultiValueReader<FixedBitMultiValueReader.Context> {
+public final class FixedBitMVForwardIndexReader implements ForwardIndexReader<FixedBitMVForwardIndexReader.Context> {
private static final int PREFERRED_NUM_VALUES_PER_CHUNK = 2048;
private final FixedByteValueReaderWriter _chunkOffsetReader;
private final PinotDataBitSet _bitmapReader;
private final FixedBitIntReaderWriter _rawDataReader;
- private final int _numRows;
+ private final int _numDocs;
private final int _numValues;
- private final int _numRowsPerChunk;
+ private final int _numDocsPerChunk;
- public FixedBitMultiValueReader(PinotDataBuffer dataBuffer, int numRows, int numValues, int numBitsPerValue) {
- _numRows = numRows;
+ public FixedBitMVForwardIndexReader(PinotDataBuffer dataBuffer, int numDocs, int numValues, int numBitsPerValue) {
+ _numDocs = numDocs;
_numValues = numValues;
- _numRowsPerChunk = (int) (Math.ceil((float) PREFERRED_NUM_VALUES_PER_CHUNK / (numValues / numRows)));
- int numChunks = (numRows + _numRowsPerChunk - 1) / _numRowsPerChunk;
+ _numDocsPerChunk = (int) (Math.ceil((float) PREFERRED_NUM_VALUES_PER_CHUNK / (numValues / numDocs)));
+ int numChunks = (numDocs + _numDocsPerChunk - 1) / _numDocsPerChunk;
long endOffset = numChunks * Integer.BYTES;
_chunkOffsetReader = new FixedByteValueReaderWriter(dataBuffer.view(0L, endOffset));
int bitmapSize = (numValues + Byte.SIZE - 1) / Byte.SIZE;
@@ -71,43 +74,41 @@ public final class FixedBitMultiValueReader extends BaseSingleColumnMultiValueRe
}
@Override
- public int getIntArray(int row, int[] intArray) {
- int chunkId = row / _numRowsPerChunk;
- int chunkOffset = _chunkOffsetReader.getInt(chunkId);
- int indexInChunk = row % _numRowsPerChunk;
- int startIndex;
- if (indexInChunk == 0) {
- startIndex = chunkOffset;
- } else {
- startIndex = _bitmapReader.getNextNthSetBitOffset(chunkOffset + 1, indexInChunk);
- }
- int endIndex;
- if (row == _numRows - 1) {
- endIndex = _numValues;
- } else {
- endIndex = _bitmapReader.getNextSetBitOffset(startIndex + 1);
- }
- int numValues = endIndex - startIndex;
- _rawDataReader.readInt(startIndex, numValues, intArray);
- return numValues;
+ public boolean isDictionaryEncoded() {
+ return true;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return false;
+ }
+
+ @Override
+ public DataType getValueType() {
+ return DataType.INT;
+ }
+
+ @Override
+ public Context createContext() {
+ return new Context();
}
@Override
- public int getIntArray(int row, int[] intArray, Context context) {
- int contextRow = context._row;
+ public int getDictIdMV(int docId, int[] dictIdBuffer, Context context) {
+ int contextDocId = context._docId;
int contextEndOffset = context._endOffset;
int startIndex;
- if (row == contextRow + 1) {
+ if (docId == contextDocId + 1) {
startIndex = contextEndOffset;
} else {
- int chunkId = row / _numRowsPerChunk;
- if (row > contextRow && chunkId == contextRow / _numRowsPerChunk) {
+ int chunkId = docId / _numDocsPerChunk;
+ if (docId > contextDocId && chunkId == contextDocId / _numDocsPerChunk) {
// Same chunk
- startIndex = _bitmapReader.getNextNthSetBitOffset(contextEndOffset + 1, row - contextRow - 1);
+ startIndex = _bitmapReader.getNextNthSetBitOffset(contextEndOffset + 1, docId - contextDocId - 1);
} else {
// Different chunk
int chunkOffset = _chunkOffsetReader.getInt(chunkId);
- int indexInChunk = row % _numRowsPerChunk;
+ int indexInChunk = docId % _numDocsPerChunk;
if (indexInChunk == 0) {
startIndex = chunkOffset;
} else {
@@ -116,27 +117,22 @@ public final class FixedBitMultiValueReader extends BaseSingleColumnMultiValueRe
}
}
int endIndex;
- if (row == _numRows - 1) {
+ if (docId == _numDocs - 1) {
endIndex = _numValues;
} else {
endIndex = _bitmapReader.getNextSetBitOffset(startIndex + 1);
}
int numValues = endIndex - startIndex;
- _rawDataReader.readInt(startIndex, numValues, intArray);
+ _rawDataReader.readInt(startIndex, numValues, dictIdBuffer);
// Update context
- context._row = row;
+ context._docId = docId;
context._endOffset = endIndex;
return numValues;
}
@Override
- public Context createContext() {
- return new Context();
- }
-
- @Override
public void close() {
// NOTE: DO NOT close the PinotDataBuffer here because it is tracked by the caller and might be reused later. The
// caller is responsible of closing the PinotDataBuffer.
@@ -145,9 +141,13 @@ public final class FixedBitMultiValueReader extends BaseSingleColumnMultiValueRe
_rawDataReader.close();
}
- public static class Context implements ReaderContext {
- public int _row = -1;
+ public static class Context implements ForwardIndexReaderContext {
+ private int _docId = -1;
// Exclusive
- public int _endOffset = 0;
+ private int _endOffset = 0;
+
+ @Override
+ public void close() {
+ }
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedBitSVForwardIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedBitSVForwardIndexReader.java
new file mode 100644
index 0000000..1527cbc
--- /dev/null
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedBitSVForwardIndexReader.java
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.index.readers.forward;
+
+import org.apache.pinot.core.io.util.FixedBitIntReaderWriter;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
+import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+
+
+/**
+ * Bit-compressed dictionary-encoded forward index reader for single-value columns. The values returned are dictionary
+ * ids.
+ */
+public final class FixedBitSVForwardIndexReader implements ForwardIndexReader<ForwardIndexReaderContext> {
+ private final FixedBitIntReaderWriter _reader;
+
+ public FixedBitSVForwardIndexReader(PinotDataBuffer dataBuffer, int numDocs, int numBitsPerValue) {
+ _reader = new FixedBitIntReaderWriter(dataBuffer, numDocs, numBitsPerValue);
+ }
+
+ @Override
+ public boolean isDictionaryEncoded() {
+ return true;
+ }
+
+ @Override
+ public boolean isSingleValue() {
+ return true;
+ }
+
+ @Override
+ public DataType getValueType() {
+ return DataType.INT;
+ }
+
+ @Override
+ public int getDictId(int docId, ForwardIndexReaderContext context) {
+ return _reader.readInt(docId);
+ }
+
+ @Override
+ public void readDictIds(int[] docIds, int length, int[] dictIdBuffer, ForwardIndexReaderContext context) {
+ for (int i = 0; i < length; i++) {
+ dictIdBuffer[i] = _reader.readInt(docIds[i]);
+ }
+ }
+
+ @Override
+ public void close() {
+ _reader.close();
+ }
+}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java
new file mode 100644
index 0000000..a02ea31
--- /dev/null
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/FixedByteChunkSVForwardIndexReader.java
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.index.readers.forward;
+
+import java.nio.ByteBuffer;
+import org.apache.pinot.core.io.writer.impl.FixedByteChunkSVForwardIndexWriter;
+import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+
+
+/**
+ * Chunk-based single-value raw (non-dictionary-encoded) forward index reader for values of fixed length data type (INT,
+ * LONG, FLOAT, DOUBLE).
+ * <p>For data layout, please refer to the documentation for {@link FixedByteChunkSVForwardIndexWriter}
+ */
+public final class FixedByteChunkSVForwardIndexReader extends BaseChunkSVForwardIndexReader {
+
+ public FixedByteChunkSVForwardIndexReader(PinotDataBuffer dataBuffer, DataType valueType) {
+ super(dataBuffer, valueType);
+ }
+
+ @Override
+ public ChunkReaderContext createContext() {
+ if (_isCompressed) {
+ return new ChunkReaderContext(_chunkSize);
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public int getInt(int docId, ChunkReaderContext context) {
+ if (_isCompressed) {
+ int chunkRowId = docId % _numDocsPerChunk;
+ ByteBuffer chunkBuffer = getChunkBuffer(docId, context);
+ return chunkBuffer.getInt(chunkRowId * Integer.BYTES);
+ } else {
+ return _rawData.getInt(docId * Integer.BYTES);
+ }
+ }
+
+ @Override
+ public long getLong(int docId, ChunkReaderContext context) {
+ if (_isCompressed) {
+ int chunkRowId = docId % _numDocsPerChunk;
+ ByteBuffer chunkBuffer = getChunkBuffer(docId, context);
+ return chunkBuffer.getLong(chunkRowId * Long.BYTES);
+ } else {
+ return _rawData.getLong(docId * Long.BYTES);
+ }
+ }
+
+ @Override
+ public float getFloat(int docId, ChunkReaderContext context) {
+ if (_isCompressed) {
+ int chunkRowId = docId % _numDocsPerChunk;
+ ByteBuffer chunkBuffer = getChunkBuffer(docId, context);
+ return chunkBuffer.getFloat(chunkRowId * Float.BYTES);
+ } else {
+ return _rawData.getFloat(docId * Float.BYTES);
+ }
+ }
+
+ @Override
+ public double getDouble(int docId, ChunkReaderContext context) {
+ if (_isCompressed) {
+ int chunkRowId = docId % _numDocsPerChunk;
+ ByteBuffer chunkBuffer = getChunkBuffer(docId, context);
+ return chunkBuffer.getDouble(chunkRowId * Double.BYTES);
+ } else {
+ return _rawData.getDouble(docId * Double.BYTES);
+ }
+ }
+}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/VarByteChunkSingleValueReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java
similarity index 61%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/VarByteChunkSingleValueReader.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java
index 418a0f3..e0785d8 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/VarByteChunkSingleValueReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/forward/VarByteChunkSVForwardIndexReader.java
@@ -16,46 +16,44 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.reader.impl.v1;
+package org.apache.pinot.core.segment.index.readers.forward;
import java.nio.ByteBuffer;
import org.apache.pinot.common.utils.StringUtil;
-import org.apache.pinot.core.io.reader.impl.ChunkReaderContext;
-import org.apache.pinot.core.io.writer.impl.v1.VarByteChunkSingleValueWriter;
+import org.apache.pinot.core.io.writer.impl.VarByteChunkSVForwardIndexWriter;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
/**
- * Reader class for data written out by {@link VarByteChunkSingleValueWriter}.
- * For data layout, please refer to the documentation for {@link VarByteChunkSingleValueWriter}
+ * Chunk-based single-value raw (non-dictionary-encoded) forward index reader for values of of variable length data
+ * type (STRING, BYTES).
+ * <p>For data layout, please refer to the documentation for {@link VarByteChunkSVForwardIndexWriter}
*/
-public class VarByteChunkSingleValueReader extends BaseChunkSingleValueReader {
+public final class VarByteChunkSVForwardIndexReader extends BaseChunkSVForwardIndexReader {
private final int _maxChunkSize;
// Thread local (reusable) byte[] to read bytes from data file.
private final ThreadLocal<byte[]> _reusableBytes = ThreadLocal.withInitial(() -> new byte[_lengthOfLongestEntry]);
- /**
- * Constructor for the class.
- *
- * @param pinotDataBuffer Data buffer to read from
- */
- public VarByteChunkSingleValueReader(PinotDataBuffer pinotDataBuffer) {
- super(pinotDataBuffer);
- _maxChunkSize = _numDocsPerChunk * (VarByteChunkSingleValueWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE + _lengthOfLongestEntry);
+ public VarByteChunkSVForwardIndexReader(PinotDataBuffer dataBuffer, DataType valueType) {
+ super(dataBuffer, valueType);
+ _maxChunkSize = _numDocsPerChunk * (VarByteChunkSVForwardIndexWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE
+ + _lengthOfLongestEntry);
}
@Override
- public String getString(int row) {
- return getString(row, createContext());
+ public ChunkReaderContext createContext() {
+ return new ChunkReaderContext(_maxChunkSize);
}
@Override
- public String getString(int row, ChunkReaderContext context) {
- int chunkRowId = row % _numDocsPerChunk;
- ByteBuffer chunkBuffer = getChunkForRow(row, context);
+ public String getString(int docId, ChunkReaderContext context) {
+ int chunkRowId = docId % _numDocsPerChunk;
+ ByteBuffer chunkBuffer = getChunkBuffer(docId, context);
- int rowOffset = chunkBuffer.getInt(chunkRowId * VarByteChunkSingleValueWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE);
+ int rowOffset =
+ chunkBuffer.getInt(chunkRowId * VarByteChunkSVForwardIndexWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE);
int nextRowOffset = getNextRowOffset(chunkRowId, chunkBuffer);
int length = nextRowOffset - rowOffset;
@@ -68,16 +66,12 @@ public class VarByteChunkSingleValueReader extends BaseChunkSingleValueReader {
}
@Override
- public byte[] getBytes(int row) {
- return getBytes(row, createContext());
- }
+ public byte[] getBytes(int docId, ChunkReaderContext context) {
+ int chunkRowId = docId % _numDocsPerChunk;
+ ByteBuffer chunkBuffer = getChunkBuffer(docId, context);
- @Override
- public byte[] getBytes(int row, ChunkReaderContext context) {
- int chunkRowId = row % _numDocsPerChunk;
- ByteBuffer chunkBuffer = getChunkForRow(row, context);
-
- int rowOffset = chunkBuffer.getInt(chunkRowId * VarByteChunkSingleValueWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE);
+ int rowOffset =
+ chunkBuffer.getInt(chunkRowId * VarByteChunkSVForwardIndexWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE);
int nextRowOffset = getNextRowOffset(chunkRowId, chunkBuffer);
int length = nextRowOffset - rowOffset;
@@ -88,11 +82,6 @@ public class VarByteChunkSingleValueReader extends BaseChunkSingleValueReader {
return bytes;
}
- @Override
- public ChunkReaderContext createContext() {
- return new ChunkReaderContext(_maxChunkSize);
- }
-
/**
* Helper method to compute the offset of next row in the chunk buffer.
*
@@ -109,7 +98,8 @@ public class VarByteChunkSingleValueReader extends BaseChunkSingleValueReader {
// Last row in this trunk.
nextRowOffset = chunkBuffer.limit();
} else {
- nextRowOffset = chunkBuffer.getInt((currentRowId + 1) * VarByteChunkSingleValueWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE);
+ nextRowOffset =
+ chunkBuffer.getInt((currentRowId + 1) * VarByteChunkSVForwardIndexWriter.CHUNK_HEADER_ENTRY_ROW_OFFSET_SIZE);
// For incomplete chunks, the next string's offset will be 0 as row offset for absent rows are 0.
if (nextRowOffset == 0) {
nextRowOffset = chunkBuffer.limit();
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/SortedIndexReaderImpl.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/sorted/SortedIndexReaderImpl.java
similarity index 63%
rename from pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/SortedIndexReaderImpl.java
rename to pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/sorted/SortedIndexReaderImpl.java
index 50cac18..605c1a4 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/io/reader/impl/v1/SortedIndexReaderImpl.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/sorted/SortedIndexReaderImpl.java
@@ -16,18 +16,21 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.io.reader.impl.v1;
+package org.apache.pinot.core.segment.index.readers.sorted;
import com.google.common.base.Preconditions;
import java.io.IOException;
import org.apache.pinot.common.utils.Pairs;
-import org.apache.pinot.core.io.reader.BaseSingleColumnSingleValueReader;
-import org.apache.pinot.core.io.reader.ReaderContext;
import org.apache.pinot.core.io.util.FixedByteValueReaderWriter;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
+import org.apache.pinot.core.segment.index.readers.SortedIndexReader;
import org.apache.pinot.core.segment.memory.PinotDataBuffer;
-public class SortedIndexReaderImpl extends BaseSingleColumnSingleValueReader<SortedIndexReaderImpl.Context> implements SortedIndexReader<SortedIndexReaderImpl.Context> {
+/**
+ * Implementation of sorted index reader.
+ */
+public class SortedIndexReaderImpl implements SortedIndexReader<SortedIndexReaderImpl.Context> {
private final FixedByteValueReaderWriter _reader;
private final int _cardinality;
@@ -39,17 +42,12 @@ public class SortedIndexReaderImpl extends BaseSingleColumnSingleValueReader<Sor
}
@Override
- public int getInt(int row) {
- // Only one value in dictionary
- if (_cardinality == 1) {
- return 0;
- }
-
- return binarySearch(row, 0, _cardinality - 1);
+ public Context createContext() {
+ return new Context();
}
@Override
- public int getInt(int row, Context context) {
+ public int getDictId(int docId, Context context) {
// Only one value in dictionary
if (_cardinality == 1) {
return 0;
@@ -58,16 +56,16 @@ public class SortedIndexReaderImpl extends BaseSingleColumnSingleValueReader<Sor
int contextDictId = context._dictId;
int contextStartOffset = context._startOffset;
int contextEndOffset = context._endOffset;
- if (row >= contextStartOffset) {
+ if (docId >= contextStartOffset) {
// Same value
- if (row <= contextEndOffset) {
+ if (docId <= contextEndOffset) {
return contextDictId;
}
// Next value in dictionary
int nextDictId = contextDictId + 1;
int nextEndOffset = _reader.getInt(2 * nextDictId + 1);
- if (row <= nextEndOffset) {
+ if (docId <= nextEndOffset) {
context._dictId = nextDictId;
context._startOffset = contextEndOffset + 1;
context._endOffset = nextEndOffset;
@@ -76,10 +74,10 @@ public class SortedIndexReaderImpl extends BaseSingleColumnSingleValueReader<Sor
}
int dictId;
- if (row < contextStartOffset) {
- dictId = binarySearch(row, 0, contextDictId - 1);
+ if (docId < contextStartOffset) {
+ dictId = binarySearch(docId, 0, contextDictId - 1);
} else {
- dictId = binarySearch(row, contextDictId + 2, _cardinality - 1);
+ dictId = binarySearch(docId, contextDictId + 2, _cardinality - 1);
}
context._dictId = dictId;
context._startOffset = _reader.getInt(2 * dictId);
@@ -100,26 +98,19 @@ public class SortedIndexReaderImpl extends BaseSingleColumnSingleValueReader<Sor
}
@Override
- public void readValues(int[] rows, int rowsStartIndex, int rowSize, int[] values, int valuesStartIndex) {
- int rowsEndIndex = rowsStartIndex + rowSize;
+ public void readDictIds(int[] docIds, int length, int[] dictIdBuffer, Context context) {
if (_cardinality == 1) {
- for (int i = rowsStartIndex; i < rowsEndIndex; i++) {
- values[valuesStartIndex++] = 0;
+ for (int i = 0; i < length; i++) {
+ dictIdBuffer[i] = 0;
}
} else {
- Context context = new Context();
- for (int i = rowsStartIndex; i < rowsEndIndex; i++) {
- values[valuesStartIndex++] = getInt(rows[i], context);
+ for (int i = 0; i < length; i++) {
+ dictIdBuffer[i] = getDictId(docIds[i], context);
}
}
}
@Override
- public Context createContext() {
- return new Context();
- }
-
- @Override
public Pairs.IntPair getDocIds(int dictId) {
return new Pairs.IntPair(_reader.getInt(2 * dictId), _reader.getInt(2 * dictId + 1));
}
@@ -130,17 +121,14 @@ public class SortedIndexReaderImpl extends BaseSingleColumnSingleValueReader<Sor
_reader.close();
}
- public static class Context implements ReaderContext {
- public int _dictId = -1;
- public int _startOffset = -1;
+ public static class Context implements ForwardIndexReaderContext {
+ private int _dictId = -1;
+ private int _startOffset = -1;
// Inclusive
- public int _endOffset = -1;
- }
+ private int _endOffset = -1;
- @Override
- public Pairs.IntPair getDocIds(Object value) {
- // This should not be called from anywhere. If it happens, there is a bug
- // and that's why we throw illegal state exception
- throw new IllegalStateException("sorted inverted index reader supports lookup only on dictionary id");
+ @Override
+ public void close() {
+ }
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java
index fa48b0d..f4b936c 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java
@@ -77,7 +77,8 @@ public class LuceneTextIndexReader implements InvertedIndexReader<MutableRoaring
_indexDirectory = FSDirectory.open(indexFile.toPath());
_indexReader = DirectoryReader.open(_indexDirectory);
_indexSearcher = new IndexSearcher(_indexReader);
- if (textIndexProperties == null || !Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_ENABLE_QUERY_CACHE))) {
+ if (textIndexProperties == null || !Boolean
+ .parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_ENABLE_QUERY_CACHE))) {
// Disable Lucene query result cache. While it helps a lot with performance for
// repeated queries, on the downside it cause heap issues.
_indexSearcher.setQueryCache(null);
@@ -122,14 +123,8 @@ public class LuceneTextIndexReader implements InvertedIndexReader<MutableRoaring
throw new IllegalStateException("Using dictionary ID is not supported on Lucene inverted index");
}
- /**
- * Get docIds from the text inverted index for a given raw value
- * @param value value to look for in the inverted index
- * @return docIDs in bitmap
- */
@Override
- public MutableRoaringBitmap getDocIds(Object value) {
- String searchQuery = (String) value;
+ public MutableRoaringBitmap getDocIds(String searchQuery) {
MutableRoaringBitmap docIds = new MutableRoaringBitmap();
Collector docIDCollector = new LuceneDocIdCollector(docIds, _docIdTranslator);
try {
@@ -141,7 +136,8 @@ public class LuceneTextIndexReader implements InvertedIndexReader<MutableRoaring
_indexSearcher.search(query, docIDCollector);
return docIds;
} catch (Exception e) {
- String msg = "Caught excepttion while searching the text index for column:" + _column + " search query:" + searchQuery;
+ String msg =
+ "Caught excepttion while searching the text index for column:" + _column + " search query:" + searchQuery;
throw new RuntimeException(msg, e);
}
}
@@ -195,7 +191,8 @@ public class LuceneTextIndexReader implements InvertedIndexReader<MutableRoaring
int pinotDocId = Integer.parseInt(document.get(LuceneTextIndexCreator.LUCENE_INDEX_DOC_ID_COLUMN_NAME));
_buffer.putInt(i * Integer.BYTES, pinotDocId);
} catch (Exception e) {
- throw new RuntimeException("Caught exception while building doc id mapping for text index column: " + column, e);
+ throw new RuntimeException(
+ "Caught exception while building doc id mapping for text index column: " + column, e);
}
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/virtualcolumn/DocIdVirtualColumnProvider.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/virtualcolumn/DocIdVirtualColumnProvider.java
index a0bc0f2..eb6624e 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/virtualcolumn/DocIdVirtualColumnProvider.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/virtualcolumn/DocIdVirtualColumnProvider.java
@@ -20,26 +20,25 @@ package org.apache.pinot.core.segment.virtualcolumn;
import java.io.IOException;
import org.apache.pinot.common.utils.Pairs;
-import org.apache.pinot.core.io.reader.BaseSingleColumnSingleValueReader;
-import org.apache.pinot.core.io.reader.DataFileReader;
-import org.apache.pinot.core.io.reader.impl.ChunkReaderContext;
-import org.apache.pinot.core.io.reader.impl.v1.SortedIndexReader;
-import org.apache.pinot.core.io.reader.impl.v1.SortedIndexReaderImpl;
import org.apache.pinot.core.segment.index.column.BaseVirtualColumnProvider;
import org.apache.pinot.core.segment.index.metadata.ColumnMetadata;
import org.apache.pinot.core.segment.index.readers.Dictionary;
import org.apache.pinot.core.segment.index.readers.DocIdDictionary;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReader;
+import org.apache.pinot.core.segment.index.readers.ForwardIndexReaderContext;
... 4452 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org